How to fix OOMKilled (exit code 137) in Kubernetes

OOMKilled (exit code 137) means the Linux kernel's out-of-memory (OOM) killer terminated your container because it exceeded its memory limit. Kubernetes sets the limit; the kernel enforces it. To fix it: identify which container was killed, verify the memory limit is correctly set, and either increase the limit or reduce the application's memory footprint.

How to confirm OOMKilled

Start by confirming the container actually died from an OOM kill and not another reason.

kubectl describe pod <pod-name> -n <namespace>

In the output, look for the container's last state:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Exit code 137 is the standard signal: the process received SIGKILL (signal 9) from the kernel's OOM killer. You can also check cluster events for additional context:

kubectl get events -n <namespace> --sort-by='.lastTimestamp' | grep -i oom

Look for events with reason OOMKilling. These appear on the node, so you may need to check events without a namespace filter if nothing appears:

kubectl get events --all-namespaces --sort-by='.lastTimestamp' | grep -i oom

Find the offending container

Pods often run multiple containers. Narrow down which one consumed the memory.

kubectl top pod <pod-name> --containers -n <namespace>

This requires metrics-server to be installed. The output shows current memory usage per container. Compare each container's current usage against its configured limit. The container closest to (or exceeding) its limit is the likely culprit.

To see the configured limits:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.limits.memory}{"\n"}{end}'

If metrics-server is not available, check your monitoring platform (Prometheus, Datadog, CloudWatch) for historical memory usage on that pod and container.

Root causes

OOMKilled has four common root causes. The fix depends on which one applies.

1. Memory limit too low. The application's normal working set simply exceeds the configured limit. This is common when limits were set arbitrarily (e.g., copied from another service) rather than measured from actual usage.

2. Memory leak. The application allocates memory over time and never releases it. Usage grows linearly until the limit is hit. Every restart buys temporary relief; the container is killed again hours or days later.

3. Traffic spike. The workload scaled in request volume but not in resources. A sudden burst of concurrent requests caused a one-time spike that crossed the limit. Usage was fine under normal load.

4. JVM heap misconfiguration. Java applications are a common source of unexpected OOM kills. The JVM reads the container's memory limit and uses it as the basis for heap sizing, but then allocates additional off-heap memory (metaspace, direct buffers, thread stacks, JIT cache). The total JVM footprint exceeds the container limit even when the heap itself appears healthy.

Diagnostic steps

Follow this sequence to identify which root cause applies before making changes.

Step 1: Get the last restart time.

kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Last State"

Note the exact time of the OOM kill. You need this to anchor your memory usage query.

Step 2: Check the memory usage trend in the 30 minutes before the kill.

In Prometheus or your monitoring platform, query memory working set for the container:

container_memory_working_set_bytes{pod="<pod-name>", container="<container-name>"}

Look at the shape of the graph leading up to the kill.

Step 3: Distinguish a leak from a spike.

• Linear, upward growth over hours or days with no corresponding traffic increase: memory leak.
• Flat baseline with a sharp vertical spike at the moment of kill: traffic burst or a single large allocation.
• Gradual step-up that correlates with higher traffic: the limit is too low for the actual working set.

Step 4: Check limits and requests together.

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{range .spec.containers[*]}{.name}{"\n  requests.memory: "}{.resources.requests.memory}{"\n  limits.memory:   "}{.resources.limits.memory}{"\n"}{end}'

If requests and limits are very different (e.g., requests 128Mi, limits 512Mi), the pod is in the Burstable QoS class. Under memory pressure on the node, Burstable pods are the first candidates for eviction, which can compound OOM issues.

Fixes

Increase the limit (when the limit is genuinely too low).

Edit the Deployment or StatefulSet:

kubectl edit deployment <deployment-name> -n <namespace>

Update the container's resources block:

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"

Base the new limit on the observed peak usage plus at least 20% headroom. Avoid guessing.

Set requests equal to limits (Guaranteed QoS class).

When requests.memory == limits.memory, Kubernetes assigns the pod to the Guaranteed QoS class. These pods are the last to be evicted under node memory pressure and receive more predictable scheduling. For stateful services and anything latency-sensitive, Guaranteed QoS is the correct default.

Fix the leak.

For Go applications, use pprof to capture a heap profile:

go tool pprof http://<pod-ip>:6060/debug/pprof/heap

For Java applications, trigger a heap dump:

kubectl exec <pod-name> -n <namespace> -- jmap -dump:format=b,file=/tmp/heap.hprof <pid>
kubectl cp <pod-name>:/tmp/heap.hprof ./heap.hprof -n <namespace>

Analyze the dump with Eclipse MAT or VisualVM to find the allocating code path.

Fix JVM off-heap allocation (Java-specific).

Set -XX:MaxRAMPercentage instead of -Xmx. This tells the JVM to use a percentage of the container limit for the heap, leaving room for off-heap structures:

JAVA_OPTS="-XX:MaxRAMPercentage=75.0 -XX:+UseContainerSupport"

With MaxRAMPercentage=75, a container with a 1Gi limit allocates approximately 768Mi to the heap and leaves 256Mi for off-heap allocation. Tune the percentage based on your application's off-heap footprint. A general starting point is 70-75%.

Prevention

Set resource requests and limits on every container. A container without a memory limit can consume all node memory, which triggers OOM kills on neighboring pods. Use LimitRange to enforce defaults at the namespace level.

Use Vertical Pod Autoscaler (VPA) in recommendation mode. VPA watches actual usage and generates right-sizing recommendations without automatically changing your pods:

kubectl describe vpa <vpa-name> -n <namespace>

Review the Target recommendations and apply them to your manifests on your own schedule.

Alert before the kill happens. Set an alert on memory usage above 80% of the limit. By the time a container is OOMKilled, it is too late to react. An alert at 80% gives you time to investigate and resize before the next kill.

In Prometheus:

- alert: ContainerMemoryNearLimit
  expr: |
    container_memory_working_set_bytes
    / on(pod, container, namespace)
    kube_pod_container_resource_limits{resource="memory"}
    > 0.80
  for: 5m
  labels:
    severity: warning

For automated root-cause analysis on OOM events and other Kubernetes incidents, see the AI SRE Benchmark to understand how NOFire AI approaches signal-to-root-cause accuracy.

Related debugging guides

These failure modes are often connected. See also:

• Kubernetes pod stuck in Pending
• Debugging 502 and 503 errors
• PostgreSQL deadlock detection
• CrashLoopBackOff: more than just a bad deployment