The Invisible Journey: Tracing Packets Through the Linux Kernel with eBPF and `pwru`

Question

The modern developer's landscape is a complex tapestry of microservices, containers, and cloud infrastructure. When an application slows down, or a connection drops, the immediate culprits are often assumed to be the application code itself or the external network. But what about the invisible journey a packet takes inside the operating system? That often-opaque realm, the Linux kernel, holds the true story of network performance, packet drops, and elusive latency spikes.

For many developers, the kernel remains a mysterious black box. We interact with it daily through syscalls and network interfaces, but rarely peer into its inner workings. Traditional tools like tcpdump and netstat offer glimpses – what leaves our machine, what arrives, and what ports are open – but they often fall short when we need to understand why a packet was dropped, where it spent its time, or how it was processed before it ever reached the application. This is the gap that kernel-level observability tools, powered by eBPF, are rapidly filling.

Inspired by the journey of a fellow developer learning to navigate the kernel's network stack, this article aims to demystify the process of tracking a packet. We'll explore why deep kernel insight is crucial for modern debugging, introduce the revolutionary technology of eBPF, and then dive hands-on into pwru – an eBPF-based tool that makes the invisible journey of a packet through the Linux kernel visible and understandable. You don't need to be a kernel developer to unlock these insights; you just need the right tools and a guide to interpret their output.

The Invisible Journey: Why Deep Packet Tracing Matters

Imagine a web request taking too long. Your application logs show the request was received, but the response was delayed. Is it the database? The application logic? Or something happening deep within the network stack of your server, or even the client? Without kernel-level visibility, you're often left guessing.

Beyond the Wire: The Limitations of Userspace Tools

Tools like tcpdump are invaluable. They capture packets as they hit or leave a network interface, showing you what's on the wire. netstat or ss show you active connections, listening ports, and basic statistics. These are the frontline tools for network debugging, and they serve a critical purpose.

However, they have inherent limitations when you need to understand the internal processing of a packet:

• Black Box Kernel: tcpdump sees packets before the kernel processes them on ingress and after the kernel has finished processing them on egress. What happens in between – routing decisions, firewall rules, socket buffer management, TCP stack processing – remains a mystery.
• Packet Drops: If tcpdump doesn't show a packet leaving your machine, you know it never made it to the wire. But why? Was it dropped by a firewall? A full transmit queue? A routing error? tcpdump can't tell you the precise point of failure within the kernel.
• Latency Attribution: A high round-trip time could be due to network latency, but it could also be due to significant processing delays within the kernel's network stack or the application's inability to read from the socket quickly enough. Isolating these factors is hard with userspace tools alone.
• Context: Userspace tools often lack the rich context available inside the kernel, such as the specific kernel function being executed, the CPU core involved, or the network namespace.

Common Scenarios Demanding Kernel-Level Insight

Developers frequently encounter scenarios where kernel-level packet tracing becomes indispensable:

• Mysterious Packet Drops: "My service isn't receiving all the packets it expects, but tcpdump on the interface shows them arriving." This suggests an internal kernel drop.
• Unexplained Latency: "My application is slow, but CPU and memory usage are fine, and network latency to other services seems normal." Could it be contention within the kernel's network buffers or inefficient processing?
• Firewall Troubleshooting: "I've configured iptables, but traffic still isn't flowing as expected, or is being unexpectedly blocked." Pinpointing the exact netfilter hook where a packet is dropped or modified is crucial.
• Routing and Forwarding Issues: "Why isn't this packet being forwarded to the correct destination, even though my routing table looks correct?" Tracing the packet's path through the routing decision points can reveal the answer.
• Complex Network Stacks: In environments using technologies like IPVS, BPF-based load balancers, or advanced container networking (e.g., Cilium, Calico), understanding the kernel's internal packet flow becomes paramount.

These challenges highlight the need for a deeper level of observability – one that allows us to peek inside the kernel and follow a packet's every step.

Enter eBPF: Programmable Kernel Observability

For decades, getting insight into the Linux kernel involved recompiling it, using ftrace, or relying on limited procfs entries. These methods were often cumbersome, risky, or lacked the necessary detail. The advent of eBPF (extended Berkeley Packet Filter) changed everything.

What is eBPF? (A High-Level Overview)

At its core, eBPF is a revolutionary technology that allows programs to run safely within the Linux kernel. Think of it as a highly efficient, event-driven virtual machine embedded directly in the kernel. Unlike traditional kernel modules, eBPF programs are:

• Safe: All eBPF programs must pass a rigorous kernel verifier before execution, ensuring they don't crash the kernel, loop indefinitely, or access unauthorized memory.
• Event-Driven: eBPF programs are attached to various "hook points" within the kernel. These can be:
  • Kprobes/Kretprobes: Attach to the entry or exit of almost any kernel function.
  • Uprobes/Uretprobes: Attach to the entry or exit of userspace functions.
  • Tracepoints: Stable, well-defined points intentionally added by kernel developers for tracing.
  • Network Hooks: For filtering and processing network packets.
  • Syscall Hooks: Intercepting system calls.
• Efficient: They execute in-kernel, avoiding costly context switches between userspace and kernel space. They are typically JIT-compiled to native machine code for maximum performance.
• Programmable: Developers write eBPF programs (often in C, then compiled to eBPF bytecode using clang and llvm) to perform specific tasks: filter events, collect statistics, modify data, or redirect packets. These programs can then communicate results back to userspace via shared maps or ring buffers.

In essence, eBPF provides a powerful, flexible, and safe way to extend kernel functionality and gain unprecedented visibility into its operations without modifying kernel source code or rebooting.

The Power of eBPF for Networking

eBPF's origins are deeply rooted in networking (the original BPF was for packet filtering). Today, its capabilities for network observability and control are immense:

• Advanced Packet Filtering: Beyond what iptables can do, eBPF programs can filter packets based on arbitrary criteria, even deep into application-layer protocols.
• Traffic Control: Shaping, policing, and load balancing traffic directly in the kernel.
• Security: Implementing fine-grained network policies and intrusion detection.
• Observability: This is where pwru shines. By attaching eBPF programs to key network functions and tracepoints, we can collect detailed information about every packet as it traverses the kernel's network stack. This allows us to see exactly what the kernel is doing with each packet, providing unparalleled debugging capabilities.

pwru: Your Guide Through the Kernel Network Stack

While you could write your own eBPF programs to trace packets, tools like pwru (Packet Walk Ruler) abstract away the complexity, providing a ready-to-use solution for deep network stack inspection. pwru leverages eBPF to dynamically attach to numerous kernel functions and tracepoints, giving you a real-time, step-by-step view of a packet's journey.

Introducing pwru

pwru isn't just another tcpdump. It's a specialized tool designed to show you:

• Hook Point: The specific kernel function or tracepoint the packet is currently passing through.
• Action: What the kernel did with the packet at that point (e.g., PASS, CONSUMED, DROP, QUEUE).
• Packet Metadata: Crucial details like source/destination IP/port, protocol, packet length (skb_len), network namespace, CPU, and more.
• Context: Which netfilter hook, if any, was involved, and the return code.

This level of detail allows you to precisely identify where packets are being handled, modified, or dropped within the kernel, making it an indispensable tool for debugging complex network issues.

Setting Up pwru (Practical Steps)

pwru requires a relatively recent Linux kernel (4.9+ for basic eBPF, 5.x+ for full features and stability) and a few development tools to compile its eBPF programs.

1. Prerequisites:

   • Linux Kernel: Version 4.9+ (5.x or newer recommended).
   • Build Tools: clang, llvm, make, git.
   • Kernel Headers: Ensure your kernel headers match your running kernel version. On Debian/Ubuntu, sudo apt install linux-headers-$(uname -r). On Fedora/CentOS/RHEL, sudo yum install kernel-devel.

2. Installation:
   pwru is typically installed by cloning its GitHub repository and building it.

   # Ensure you have git, clang, llvm, and make installed
   sudo apt update && sudo apt install git clang llvm make   # For Debian/Ubuntu
   # sudo yum install git clang llvm make                   # For Fedora/CentOS/RHEL
   
   git clone https://github.com/cilium/pwru.git
   cd pwru
   make
   sudo make install
   

   If make install fails or you prefer not to install globally, you can run pwru directly from the pwru/bin directory.

Your First Packet Trace with pwru

Let's start with a simple example: tracing an HTTP request to a local web server (or any external website).

Open two terminal windows. In the first, start pwru:

# Terminal 1: Start pwru
# Trace TCP packets on port 80 (HTTP) or 443 (HTTPS)
# We'll customize output fields for clarity
sudo pwru --output-fields=src_ip,dst_ip,proto,port,func,action,skb_len \
          --filter "proto == IPPROTO_TCP && (dst_port == 80 || dst_port == 443)"

In the second terminal, make an HTTP request:

# Terminal 2: Make an HTTP request
curl -v http://example.com
# Or, if you have a local web server:
# curl -v http://localhost

Back in Terminal 1, you'll see a flood of output. Let's break down a typical line:

src_ip=192.168.1.100 dst_ip=93.184.216.34 proto=TCP port=51234->80 func=ip_rcv action=PASS skb_len=60
src_ip=192.168.1.100 dst_ip=93.184.216.34 proto=TCP port=51234->80 func=ip_local_deliver_finish action=PASS skb_len=60
src_ip=192.168.1.100 dst_ip=93.184.216.34 proto=TCP port=51234->80 func=tcp_v4_rcv action=PASS skb_len=60
... (many more lines) ...

What are we seeing? Each line represents a "hook point" – a specific kernel function or tracepoint – that the packet traversed.

• src_ip, dst_ip, proto, port: Basic network tuple identifying the packet.
• func: The kernel function name (ip_rcv, tcp_v4_rcv, etc.). This is the heart of pwru's insight.
• action: What happened at that function. PASS means it continued its journey. DROP would indicate it was discarded. CONSUMED often means it was processed and removed from the general network path (e.g., by the TCP stack).
• skb_len: The size of the sk_buff (socket buffer) structure carrying the packet data.

This output shows the packet moving through different layers and processing stages within the kernel. For an outgoing packet, you'd see functions like ip_output, dev_queue_xmit, etc. For an incoming packet destined for a local socket, you'd see ip_rcv, ip_local_deliver, tcp_v4_rcv, and eventually sock_queue_rcv_skb.

Deeper Dives: Understanding the Kernel Network Path

Interpreting pwru's output requires a basic understanding of how the Linux kernel processes network packets. While the full network stack is incredibly complex, we can identify key functions and their roles.

Key Kernel Hook Points and Their Significance

The kernel network stack can be broadly divided into ingress (receiving) and egress (transmitting) paths.

Ingress Path (Receiving a Packet):

1. __netif_receive_skb_core: The initial entry point for a packet received from the network driver.
2. ip_rcv: The entry point for IPv4 packets after __netif_receive_skb_core. It performs basic validation and determines if the packet is for the local host or needs forwarding.
3. ip_local_deliver_finish: If the packet is for the local host, this function is called. It passes the packet to the appropriate transport layer (TCP, UDP, ICMP).
4. tcp_v4_rcv / udp_rcv: Entry points for TCP and UDP packets, respectively, after ip_local_deliver_finish. These functions handle protocol-specific processing (e.g., TCP sequence number checks, ACK generation).
5. sock_queue_rcv_skb: The packet is finally queued to the receiving socket's buffer, making it available for a userspace application to read via recvmsg, read, etc.

Egress Path (Transmitting a Packet):

1. tcp_sendmsg / udp_sendmsg: Initiated by a userspace application's sendmsg or write call. These prepare the packet for transmission at the transport layer.
2. ip_output: Handles the IPv4 header creation and prepares the packet for sending.
3. ip_finish_output: Finalizes IP output, including routing decisions.
4. dev_queue_xmit: Queues the packet to the appropriate network device's transmit queue.
5. __dev_xmit_skb: The packet is handed off to the network driver for physical transmission.

pwru shows you the packet traversing many of these functions (and many more in between), along with netfilter hooks (NF_HOOK points) and other processing stages.

Common Scenarios & pwru Insights

Let's look at how pwru can help diagnose specific problems.

1. Pinpointing Packet Drops

One of the most powerful uses of pwru is identifying where and why a packet is dropped. If you suspect packet loss:

# Terminal 1: Watch for dropped packets
sudo pwru --output-fields=func,action,src_ip,dst_ip,proto,port,skb_len --filter "action == DROP"

Now, try to reproduce the packet loss (e.g., attempt a connection to a non-existent port on your machine, or trigger a firewall rule). You might see output like:

func=ip_rcv_finish action=DROP src_ip=192.168.1.1 dst_ip=192.168.1.100 proto=ICMP port=0->0 skb_len=84
func=nf_hook_slow action=DROP src_ip=192.168.1.1 dst_ip=192.168.1.100 proto=TCP port=12345->80 skb_len=60

• ip_rcv_finish with DROP might indicate issues like a full receive queue or an invalid packet.
• nf_hook_slow with DROP is a strong indicator that a netfilter (iptables) rule is dropping the packet. This immediately tells you to inspect your firewall configuration.

2. Analyzing Latency Attribution

If curl to localhost is slow, where is the time spent? pwru can show the timestamps for each hook point, though interpreting the exact delta requires careful analysis and often correlation with other tools. For a quick check, you can add ktime (kernel time in nanoseconds) to your output:

# Terminal 1: Trace a local request with timestamps
sudo pwru --output-fields=ktime,func,action,src_ip,dst_ip,proto,port,skb_len \
          --filter "proto == IPPROTO_TCP && (src_ip == 127.0.0.1 || dst_ip == 127.0.0.1) && (dst_port == 80 || src_port == 80)"

Then, in Terminal 2: curl http://localhost.

By examining the ktime values, you can see if there are unusually large gaps between consecutive func calls for the same packet. A long gap between ip_rcv and ip_local_deliver could suggest CPU contention or heavy processing before the packet reaches the transport layer. A delay between sock_queue_rcv_skb and the application's actual read might point to application-level slowness.

3. Understanding Firewall/Netfilter Interaction

pwru can explicitly show netfilter processing. netfilter hooks are points where iptables rules operate.

# Terminal 1: Trace packets, specifically looking for netfilter hooks
sudo pwru --output-fields=func,action,src_ip,dst_ip,proto,port,skb_len,hook_name,nf_verdict \
          --filter "proto == IPPROTO_TCP && (dst_port == 22 || src_port == 22)" # e.g., SSH traffic

You'll see lines like:

func=nf_hook_slow action=PASS hook_name=NF_INET_PRE_ROUTING nf_verdict=NF_ACCEPT src_ip=...
func=nf_hook_slow action=PASS hook_name=NF_INET_LOCAL_IN nf_verdict=NF_ACCEPT src_ip=...

• hook_name: Specifies the netfilter hook (e.g., NF_INET_PRE_ROUTING, NF_INET_LOCAL_IN, NF_INET_FORWARD).
• nf_verdict: The verdict of the netfilter chain (NF_ACCEPT, NF_DROP, NF_STOLEN, etc.).

If a packet is dropped by netfilter, you'll see an action=DROP along with nf_verdict=NF_DROP at a specific hook_name, telling you exactly which iptables chain is responsible.

4. Tracing Specific Connections

To focus on a single problematic connection, use detailed filters:

# Terminal 1: Trace a specific source IP and destination port
sudo pwru --output-fields=func,action,src_ip,dst_ip,proto,port,skb_len \
          --filter "src_ip == 10.0.0.5 && dst_port == 8080"

This ensures you only see the relevant packets, reducing noise and making analysis much easier. You can combine filters with AND, OR, NOT for very precise targeting.

Advanced pwru Techniques and Trade-offs

pwru is a powerful tool, and mastering its capabilities involves more than just basic tracing.

Filtering and Output Customization

• Complex Filters: pwru supports complex logical expressions in its --filter argument.

  # Trace HTTP and HTTPS traffic, but only from a specific source IP, and not if it's an ICMP packet
  sudo pwru --filter "proto == IPPROTO_TCP && (dst_port == 80 || dst_port == 443) && src_ip == 192.168.1.10 AND NOT proto == IPPROTO_ICMP"
  

• Custom Output Fields: The --output-fields argument is crucial. You can add fields like:

  • cpu: Which CPU core processed the packet. Useful for identifying CPU affinity issues.
  • ktime: Kernel timestamp (nanoseconds).
  • netns: Network namespace ID. Essential in containerized environments.
  • ifindex: Network interface index.
  • mark: skb->mark value, often used for policy routing or firewall marking.
  • tstamp: Hardware timestamp.
  • ingress_ifindex, egress_ifindex: The interface the packet entered/exited.
  • reason: Provides more context for drops.

  Consult pwru --help for a full list of available fields.

• Saving Output: Redirect pwru's output to a file for later analysis, especially for long-running traces.

  sudo pwru --output-fields=... --filter "..." > pwru_trace.log
  

Performance Considerations

eBPF is designed for low overhead, but pwru attaches to many kernel hook points. On a very busy system with high packet rates, tracing every packet can still generate a substantial amount of data and consume CPU cycles.

• Precision is Key: Always use the most precise filters possible. Instead of --filter "proto == IPPROTO_TCP", use proto == IPPROTO_TCP && (src_port == 12345 || dst_port == 8080) to focus on relevant traffic.
• Output Fields: Only request the output fields you truly need. Each field adds a small amount of overhead.
• Short Bursts: For initial diagnosis, run pwru for short bursts (e.g., 5-10 seconds) to capture a representative sample of traffic during a problem occurrence, rather than letting it run indefinitely.
• Production Caution: While generally safe, running pwru on highly loaded production systems without careful filtering and monitoring should be approached with caution. Test your filters in a staging environment first.

Limitations and Alternatives

While pwru is incredibly powerful, it's not a silver bullet:

• Kernel Knowledge Still Required: Interpreting the output, especially the func names, still requires some understanding of the kernel's network stack. You won't become a kernel expert overnight, but you'll learn key functions relevant to your debugging tasks.
• Root Privileges: pwru requires root privileges to attach eBPF programs to the kernel.
• Kernel Version Dependency: While pwru tries to be compatible, specific hook points or data structures can vary slightly between kernel versions, potentially affecting output or functionality.
• Not a Packet Capture Tool: pwru shows kernel events, not the full packet payload. For payload inspection, tcpdump or Wireshark are still necessary.

Alternatives and Complementary Tools:

• BCC Tools: The BPF Compiler Collection (BCC) provides a rich set of eBPF-based tools for various aspects of kernel observability, including networking (e.g., tcplife, dropwatch, sockstat). They are often more specialized than pwru.
• bpftrace: If pwru's predefined tracing isn't enough, bpftrace allows you to write custom eBPF one-liners or scripts with a simpler syntax than raw C. This is for when you need to combine specific kernel events or perform custom aggregations.
• tracepoint-analyzer: Another tool specifically designed to help understand kernel tracepoints.

pwru serves as an excellent starting point and often provides enough detail for most network debugging scenarios. When you hit its limits, bpftrace or specialized BCC tools are the next logical step.

Conclusion

The journey of a packet through the Linux kernel is often an invisible one, a black box that can hide the root causes of frustrating network performance issues, mysterious packet drops, and elusive latency. For too long, developers have been forced to debug these problems with limited visibility, relying on guesswork and external network tools that only show part of the picture.

eBPF has fundamentally changed this landscape, empowering us to safely and efficiently peer into the kernel's inner workings. Tools like pwru democratize this power, translating complex kernel events into actionable insights that even developers "who barely know the kernel" can understand and leverage.

By embracing pwru and its eBPF underpinnings, you gain a superpower: the ability to trace a packet's every step, understand its fate, and pinpoint precisely where things go wrong within the operating system. This deeper understanding