Question

Controlling Memory and CPU Utilization for Go Apps in Kubernetes: GOMEMLIMIT and GOMAXPROCS

While running Go apps in Kubernetes, we have to control both CPU and memory to avoid instability, performance issues, or unintentional pod restarts. Go gives us two potent environment variables to achieve this: GOMEMLIMIT and GOMAXPROCS. This article details how to use them in Kubernetes correctly, what to look out for, and production deployment best practices.

---

GOMEMLIMIT: Memory Usage Control

What is it?

GOMEMLIMIT is an environment variable introduced in Go 1.19. This specifies the maximum memory limit that the Go GC will try to remain below.

Setting it up in Kubernetes

It can also be defined directly as an environment variable:

env:
- name: GOMEMLIMIT
  value: "512MiB"

Or you can automatically bind it to the container memory limit with the Downward API:

env:
- name: GOMEMLIMIT
  valueFrom:
    resourceFieldRef:
      resource: limits.memory

You must have resources.limits.memory specified in order for this to function. Otherwise, the variable will be empty.

Full example:

apiVersion: v1
kind: Pod
metadata:
  name: go-app
spec:
  containers:
  - name: go-container
    image: your-go-image
    resources:
      limits:
        memory: "512Mi"
    env:
    - name: GOMEMLIMIT
      valueFrom:
        resourceFieldRef:
          resource: limits.memory

Caveats and Risks

• Often GC cycles: If the threshold is too narrow then GC will occur frequently and lead to higher CPU activity.
• Slower performance: Application performance slows down because GC is executed frequently.
• OOMKill still an option: In the event memory spikes quicker than GC can catch up, the container can nonetheless be terminated.
• Unbounded non-heap memory: Non-heap memory is not monitored: memory from cgo allocations, mmap allocations, or big slices is unmonitored

Recommendations

• Set GOMEMLIMIT to a little below your container’s memory limit (e.g., 450MiB if the limit is 512MiB).
• Test under load to monitor GC behavior.
• Make use of Go metrics (runtime/metrics) and pprof to examine memory use.

---

GOMAXPROCS: CPU Concurrency Control

What is it?

GOMAXPROCS determines the max number of OS threads to execute Go code concurrently. Go defaults to the number of visible CPU cores — but this can be misleading in containers.

Issues in Kubernetes

Even if your container enforces a CPU limit (e.g., 1 CPU), Go can uncover more cores (e.g., the entire host’s number of CPUs) and cause GOMAXPROCS to become too large, leading to:

• Overcommitted CPU usage
• Unfair scheduling
• Throttling by kubelet or kernel

Putting it in place

Do not hardcode GOMAXPROCS from container limits directly,

resources:
  limits:
    cpu: "0.5"

This will not directly convert to an integer and Go will need GOMAXPROCS to be an integer.

Instead, use Uber’s automaxprocs, which automatically reads the cgroup CPU limit and sets GOMAXPROCS accordingly:

import _ "go.uber.org/automaxprocs"

Risks

• CPU throttling if Go tries to use more CPUs than have been assigned.
• Non-deterministic performance when multiple Go processes are vying for limited CPU.

Recommendations

• Always include resources.limits.cpu in your pod spec.
• Auto-adjust safely with automaxprocs.
• Do not hand-tune except when absolutely necessary.

---

Can a Container Be Restarted Without Resource Limits?

Yes — a container can recover even without memory or CPU limits, as Kubernetes resumes containers due to several reasons:

1. Process crashes (non-zero exit code)
2. OOMKilled by the kernel
3. Failed liveness/readiness probes
4. Node or kubelet failures

Even if you do not set limits, a container still can potentially be OOM Killed by the kernel if the node is low on memory. Kubernetes will catch this and will restart the container (if restartPolicy: Always).

Use this command to debug:

kubectl describe pod <pod-name>

Look for:

• Last State: OOMKilled
• Exit Code
• Reason

---

Final Thoughts

By using GOMEMLIMIT and GOMAXPROCS in Kubernetes configuration, you can readily improve the stability and performance associated with your Go programs. Misusing them or not using them at all can lead to performance hotspots, program crashes, or uncontrolled use of system resources. Couple these settings with proper resource limits and observability tools to support seamless operations in production environments.

Comments

Great write-up! I've seen GOMEMLIMIT make a noticeable difference in memory stability for Go apps under sustained load, especially when paired with solid observability via runtime/metrics and performance profiling.
One question though, Have you or anyone else experimented with dynamic adjustment of GOMEMLIMIT based on actual usage patterns or external signals (e.g., HPA metrics or memory pressure events)? Wondering if there's a case for fine-tuning it beyond static values in certain workloads.

---

Boris,
Thanks for the clear explanation! How often does adjusting GOMEMLIMIT and GOMAXPROCS actually prevent OOM kills in real use?

---

In short:

GOMEMLIMIT: Limits memory usage, reducing the chance of OOM kills by giving the garbage collector a predictable target. But setting it too low can hurt performance.

GOMAXPROCS: Controls the number of CPU cores Go can use. While not directly related to memory, it can help prevent resource contention if both CPU and memory are maxed out simultaneously.

Both settings can help manage resources, but won’t fully prevent OOM kills if the app has memory leaks or is designed to use excessive memory. Profiling the app’s memory usage is key to identifying deeper issues.