Why Your Pods Keep Crashing at 2 AM: 5 Resource Limit Mistakes That Undermine Deployment Peace of Mind

It's 2:17 AM. Your phone buzzes with a PagerDuty alert: Deployment frontend-v3 — CrashLoopBackOff. You ssh into a node, run kubectl describe pod, and see OOMKilled. The pod requested 256 Mi of memory but tried to allocate 300 Mi. A classic resource limit mistake. But why does this always happen at 2 AM? And why do the same pods run fine during the day?

This guide walks through five resource limit mistakes that quietly sabotage Kubernetes deployments, especially under variable load. We'll explain the mechanisms behind each pitfall, show a worked example, and offer practical steps to build a more resilient system. If you manage container orchestration for a living, these patterns will help you sleep through the night.

Why Resource Limits Break at Night

Resource limits in Kubernetes aren't just quotas — they are contracts between the scheduler, the kubelet, and the Linux kernel. When those contracts are misconfigured, the system behaves unpredictably. The most common failure pattern is this: during low traffic, pods consume minimal resources and everything looks fine. Then a traffic spike — or a batch job — pushes memory or CPU usage over the configured limit, and the kernel steps in.

For CPU, the kernel throttles the container, slowing it down. For memory, the kernel kills the process (OOMKill). Both happen silently from the application's perspective, except for the crash. And because many teams only test under normal load, these failures surface during off-peak hours when automated jobs or unexpected traffic bursts occur. The result: a 2 AM outage that's hard to reproduce during business hours.

The compressible vs. incompressible trap

CPU is compressible — the kernel can limit usage by throttling time slices. Memory is incompressible — once a container hits its limit, it's killed. Teams often treat both the same way, setting tight limits on both without understanding the consequences. A container that is CPU-throttled may appear healthy but respond slowly, triggering timeouts and cascading failures. A container that hits its memory limit is simply dead. The mistake is setting memory limits too low based on idle measurements, ignoring that memory usage often grows with load.

Why night-time specifically?

Many organizations run nightly batch jobs, data syncs, or backup processes. These can spike resource usage on nodes that were already near capacity. Additionally, if your cluster uses cluster autoscaling, new nodes may not spin up quickly enough during a sudden load surge, leaving existing pods to compete for resources. Combine that with misconfigured limits, and you have a recipe for after-hours incidents.

Five Mistakes That Sabotage Your Deployments

After working with dozens of teams, we've seen the same resource limit mistakes recur. Each one undermines the stability you expect from an orchestrated environment. Here are the five most destructive.

Mistake 1: Setting requests equal to limits (or not setting requests at all)

Many tutorials show resources: { limits: { memory: "256Mi" } } without a request. This means the pod can be scheduled on a node with less than 256 Mi available — then when it tries to use its limit, the node may be oversubscribed. Conversely, setting requests equal to limits (a common “best practice”) guarantees your pod gets exactly what it asks for, but it wastes cluster capacity and prevents bin-packing. The right approach: requests should reflect baseline usage; limits should allow for bursts, but with headroom.

Mistake 2: Using absolute values without historical data

We've seen teams set memory limits to 512 Mi because “it sounds reasonable.” Without monitoring, you're guessing. A pod that normally uses 200 Mi may spike to 400 Mi during a cache warm-up. If your limit is 512 Mi, you're safe — but if you set it to 256 Mi, you'll get OOM kills. The fix: use tools like Prometheus or the Kubernetes Metrics Server to observe actual usage over a week, then set requests at the 50th percentile and limits at the 90th or 95th percentile.

Mistake 3: Ignoring the OS page cache

Linux uses free memory for caching disk I/O. The kernel may report high memory usage even if the application isn't actively using it. But when you set a memory limit, the kernel counts the page cache against the container's usage. A container that reads many files may exceed its limit even if its heap is small. This is especially common in Java applications with large file reads or in data-processing pods. The fix: either increase limits to account for cache, or use emptyDir with a size limit and manage cache manually.

Mistake 4: Overlooking sidecar resource consumption

Service meshes, log collectors, and monitoring agents run as sidecar containers in the same pod. Each sidecar has its own resource request and limit. If the main application is tuned perfectly but the sidecar has no limits, it can consume node resources and cause the main container to be evicted. Worse, sidecars are often forgotten during capacity planning. Always set resource requests and limits on every container in a pod, including init containers.

Mistake 5: Not accounting for node-level overhead

Kubernetes reserves resources for system daemons (kubelet, container runtime, OS). The --system-reserved and --kube-reserved flags control this. If these are not configured, the scheduler may pack pods onto a node beyond its actual capacity. When the node runs out of memory, the kubelet evicts pods based on quality of service class. Burstable pods (those with requests < limits) are evicted first. The mistake: assuming all node resources are available for pods. The fix: configure system reservations and monitor node-level pressure.

How Resource Limits Work Under the Hood

To understand why these mistakes hurt, you need to know what happens when a pod exceeds a limit. Kubernetes uses Linux cgroups v1 or v2 to enforce constraints. Each container is placed in a cgroup with CPU and memory limits.

CPU throttling

When a container exceeds its CPU limit, the kernel uses the Completely Fair Scheduler (CFS) to throttle it. The container is allowed to run only for a certain number of microseconds per period (default 100ms). If it uses its quota, it's throttled until the next period. This can cause latency spikes and timeouts. You can detect throttling by looking at container_cpu_cfs_throttled_seconds_total in Prometheus. A high throttling rate indicates limits are too low.

Memory OOM

When a container exceeds its memory limit, the kernel invokes the Out-Of-Memory (OOM) killer. The OOM killer selects a process to kill based on a score. The container's process is usually the target. The pod then restarts (if restart policy is Always) and may crash again. The event is logged as OOMKilled in the pod status. You can see it with kubectl describe pod.

Quality of Service (QoS) classes

Kubernetes assigns a QoS class to each pod based on requests and limits. Guaranteed pods have requests equal to limits for all containers. Burstable pods have at least one container with requests less than limits. BestEffort pods have no requests or limits set. When a node is under memory pressure, the kubelet evicts BestEffort pods first, then Burstable, then Guaranteed. This means a pod with no limits (BestEffort) is the first to be killed, even if it's behaving well. Setting limits incorrectly can accidentally demote a pod to a lower QoS class.

Worked Example: A Web App That Crashes at 2 AM

Let's walk through a realistic scenario. Your team runs a Node.js web application on a Kubernetes cluster. The deployment has 3 replicas, each with requests: { cpu: "100m", memory: "128Mi" } and limits: { cpu: "200m", memory: "256Mi" }. During the day, traffic is moderate, and memory usage hovers around 100 Mi. At night, a cron job runs that syncs data from an external API and writes to a local SQLite file. The sync causes memory usage to spike to 250 Mi, exceeding the 256 Mi limit. The container is OOMKilled. Kubernetes restarts it, but the sync fails midway, and the pod enters CrashLoopBackOff.

What went wrong? The memory limit was set based on daytime usage, ignoring the batch job's memory footprint. Also, the CPU limit of 200m may cause throttling during the sync, making the operation slower and increasing memory pressure. The fix: increase the memory limit to 512 Mi (with request at 256 Mi) and raise CPU limit to 500m. But that's not enough — you also need to ensure the sync doesn't run on all replicas simultaneously. Use a cronJob with a dedicated pod that has its own resource profile, or use pod anti-affinity to run the sync on a separate node.

Another layer: the sidecar running a log shipper (Fluentd) had no resource limits. During the sync, Fluentd's memory usage also spiked, contributing to node pressure. The node's kubelet evicted a Burstable pod from another deployment to free memory, causing a different service to degrade. This cascading failure is typical when resource limits are not set for all containers.

Edge Cases and Exceptions

Resource limits have nuances that can trip up even experienced teams. Here are edge cases to watch for.

Sidecar containers with high overhead

As mentioned, sidecars like Envoy or Istio proxy can consume significant memory, especially under load. Their limits should be set based on their own usage patterns, not the main container's. A common mistake is to set the sidecar's memory limit to 64 Mi and wonder why it gets OOMKilled during a TLS handshake storm. Monitor sidecars independently.

Node-pressure eviction vs. OOMKill

There are two levels of memory enforcement. The kernel OOM killer acts on a single container when it exceeds its limit. Node-pressure eviction is triggered by the kubelet when the node's overall memory is low. The kubelet evicts pods based on QoS class and usage. A pod that is within its limits but on a node with memory pressure can still be evicted. This is often confused with a resource limit problem. Check the pod's reason field: Evicted vs OOMKilled.

Init containers with tight limits

Init containers run sequentially before the main containers. They often perform setup tasks like downloading files or running migrations. If an init container has a low memory limit, it may fail, preventing the pod from starting. The pod will restart the init container until it succeeds or reaches backoff limit. Always give init containers enough resources for their peak usage, especially if they download large assets.

NUMA and CPU pinning

On multi-socket servers, CPU limits can cause NUMA (Non-Uniform Memory Access) issues. If a container is limited to a few CPU cores, it may be scheduled on a socket far from its memory, increasing latency. This is rare but can cause performance degradation that looks like a resource limit problem. Use the topologyManager feature gate if you need NUMA-aware scheduling.

Limits of Resource Limits

Resource limits are a blunt instrument. They prevent runaway containers but also introduce rigidity. Here are their inherent limitations.

Limits don't prevent noisy neighbors

If two pods on the same node have limits, they can still interfere with each other through resource contention at the OS level. For example, a pod that does heavy disk I/O can starve another pod of I/O bandwidth, even if both are within their CPU/memory limits. Resource limits only cover CPU and memory, not I/O or network. For multi-tenant clusters, consider using resource quotas, priority classes, and pod disruption budgets.

Limits can waste capacity

Setting limits too high or equal to requests reduces bin-packing efficiency. You may need more nodes, increasing cost. The trade-off is between stability and utilization. Many organizations accept some waste for predictability, but it's a conscious choice.

Limits are static

Application resource usage changes over time. A limit that works today may be too low after a code update or traffic pattern shift. Regularly review and adjust limits based on monitoring data. Tools like Vertical Pod Autoscaler (VPA) can automate this, but they require careful setup and may cause pod restarts.

Limits don't cover all failure modes

A pod can crash due to bugs, deadlocks, or external dependencies even if resources are plentiful. Resource limits are just one part of a reliable deployment. Combine them with health probes, rolling updates, and proper logging.

Reader FAQ

Should I always set requests equal to limits?
Not necessarily. Equal values give you a Guaranteed QoS class, which protects your pod from eviction under node pressure. But it wastes capacity. For critical services, it's safer to set requests and limits equal. For batch jobs or less critical workloads, you can allow bursting by setting limits higher than requests.

How do I choose initial values for requests and limits?
Start with monitoring. Run your application under realistic load and observe resource usage over at least a week. Set requests at the 50th percentile and limits at the 95th percentile. If you can't monitor, use conservative estimates (e.g., 256 Mi memory, 200m CPU) and adjust later.

What tools can help me set resource limits?
Prometheus with kube-state-metrics and cAdvisor gives detailed container metrics. The Kubernetes Metrics Server provides basic CPU/memory usage. Vertical Pod Autoscaler can recommend or automatically set requests and limits. Also, consider using a resource quota per namespace to prevent runaway deployments.

Can I set CPU limits without memory limits?
Technically yes, but it's not recommended. If you set only CPU limits, a memory leak can still OOMKill the container. Always set both CPU and memory limits for every container.

What about GPU or other resources?
Kubernetes supports extended resources like GPUs, but they work differently. You must specify them as limits only; requests are not supported for extended resources. For GPU workloads, ensure you have the appropriate device plugin installed.

Practical Takeaways for Peaceful Deployments

Resource limits are a tool, not a silver bullet. To avoid 2 AM crashes, adopt these practices:

1. Monitor first, limit second. Before setting limits, collect at least a week of resource usage data from a staging or production environment. Use Prometheus or a managed monitoring service.
2. Set requests based on baseline, limits based on peaks. Requests should reflect steady-state usage; limits should allow for bursts without overcommitting the node.
3. Don't forget sidecars and init containers. Every container in a pod needs its own resource specification. Review them during code reviews.
4. Use Vertical Pod Autoscaler in recommendation mode. VPA can suggest optimal requests and limits based on historical data. Apply its recommendations gradually.
5. Implement pod disruption budgets for critical services. This ensures that voluntary disruptions (like node drains) don't take down all replicas at once.
6. Test under load. Use tools like k6 or Locust to simulate traffic spikes and observe resource usage. Adjust limits accordingly.
7. Document your resource profiles. For each service, record its expected resource usage and the reasoning behind the chosen limits. This helps new team members and future debugging.

By understanding the mechanics of resource limits and avoiding these five common mistakes, you can build a cluster that stays stable even at 2 AM. Peace of mind comes from preparation, not luck.