Writing eBPF Probes in C for Advanced Kubernetes Observability

Question

Traditional Kubernetes observability usually relies on sidecar proxies or application-level tracing, both of which introduce CPU and memory overhead. eBPF avoids this by running sandboxed C programs directly inside the Linux kernel, enabling zero-instrumentation tracing for syscalls, network events, and file platform operations.

Tracing TCP Connect Latency

Standard pod metrics usually miss what happens deep in the networking stack. By tracing tcp_v4_connect, we can measure exact TCP connection setup times across nodes without modifying application code or container images.

The eBPF C Program

The probe below is written in restricted C and compiled into eBPF bytecode using Clang/LLVM.

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

// Define a BPF map to store the start time of the connect call, keyed by PID
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, u32);
    __type(value, u64);
} start SEC(".maps");

// Kprobe triggered on entry to tcp_v4_connect
SEC("kprobe/tcp_v4_connect")
int BPF_KPROBE(tcp_v4_connect_enter, struct sock *sk)
{
    u64 ts = bpf_ktime_get_ns();
    u32 pid = bpf_get_current_pid_tgid() >> 32;

    bpf_map_update_elem(&start, &pid, &ts, BPF_ANY);
    return 0;
}

// Kretprobe triggered on exit of tcp_v4_connect
SEC("kretprobe/tcp_v4_connect")
int BPF_KRETPROBE(tcp_v4_connect_exit, int ret)
{
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 *tsp, delta_us;

    // Lookup the start time
    tsp = bpf_map_lookup_elem(&start, &pid);
    if (!tsp) {
        return 0; // Missed entry
    }

    // Calculate latency in microseconds
    delta_us = (bpf_ktime_get_ns() - *tsp) / 1000;
    
    // bpf_printk logs to /sys/kernel/debug/tracing/trace_pipe
    bpf_printk("PID %d TCP connect took %llu us, ret = %d\n", pid, delta_us, ret);

    bpf_map_delete_elem(&start, &pid);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

Key Mechanics & Kernel Integration

• #include <vmlinux.h>: Auto-generated from kernel BTF (BPF Type Format). It provides definitions for internal kernel structs without needing multiple kernel header files.
• BPF_MAP_TYPE_HASH: eBPF programs cannot leverage global variables for dynamic state. Hash maps allow entry and exit probes to share timestamps across process IDs.
• bpf_ktime_get_ns(): Fetches high-resolution monotonic timestamps from the kernel clock.
• bpf_get_current_pid_tgid(): Returns the thread group ID in the upper 32 bits and thread ID in the lower 32 bits. Right-shifting by 32 extracts the user-space PID.
• SEC("kprobe/...") / SEC("kretprobe/..."): Macros defining ELF section names. Loaders like libbpf read these to attach probes to kernel functions.

Deploying this probe via a Kubernetes DaemonSet alongside a user-space agent (for example, written in Go using cilium/ebpf) provides low-overhead network latency tracing across all pods on a node.

---

Originally published on anasrhimi.tech.