Mastering ksniff: A Comprehensive Guide to Capturing Pod Traffic in Kubernetes with tcpdump and Wireshark

Capturing tcpdump Output in Kubernetes Pods Using ksniff

Recently, I discovered an interesting library called ksniff, which is a kubectl plugin. It utilizes tcpdump to remotely capture traffic in pods within a Kubernetes cluster, allowing you to save the data to a file or output it to Wireshark for network troubleshooting. Here’s how to use it:

Many pods actually don’t have the tcpdump executable file, so how does it remotely execute the tcpdump command in the Pods of a Kubernetes cluster? And how does it export the tcpdump output from the Pod and pass it directly to Wireshark? Below, we analyze the implementation of this tool.

ksniff has two modes of operation: privileged mode and unprivileged mode. First, let’s look at unprivileged mode.

“Unprivileged Mode in ksniff”

The logic of running in unprivileged mode is:

Upload ksniff tcpdump Executable

ksniff uses the tar command to package the tcpdump executable file, then uses the client-go library to extract it into the pod, and finally executes the tcpdump command:

The implementation of tar packaging is as follows:

Remote Command Execution Using ksniff

The following is the code for remote execution of commands in a pod. It is the standard usage of the client-go library, with nothing special:

Execute tcpdump Command

This step involves assembling the remote command and executing it in the target pod:

The Wireshark library supports input redirection. After creating an input, it can be used as a parameter for the remote tcpdump command to redirect the pod’s output to Wireshark:

“Privileged Mode with ksniff”

The implementation of privileged mode is somewhat complex. In this mode, ksniff creates a privileged pod on the node where the target pod resides (obtained through the target pod’s fields), mounts the host’s directory and the default container socket, and then calls the corresponding container runtime command within the privileged pod to execute the tcpdump command. ksniff supports three common container runtimes: containerd, cri-o, and docker. The default directories for these container runtimes are as follows:

Since privileged mode might create a new pod, it’s necessary to clean up the newly created pod after the command execution is completed.

Distinguish Container Runtimes

In privileged mode, the container runtime command on the target node is invoked. Different container runtimes have different commands, so how does ksniff distinguish between them?

ksniff uses the Kubernetes clientset to obtain the target pod’s information and determines the CRI being used through its fields. For example, the CRI is containerd, and the containerId is:

Here’s how to retrieve the container runtime and ContainerId:

Execute tcpdump Command for Different Runtimes

Let’s see how tcpdump commands are executed for different runtimes.

Containerd

Containerd pulls the tcpdump image in the privileged pod and starts the tcpdump container, making it share the same network namespace with the target container. This way, tcpdump can capture packets from the target container. After executing the command, it’s necessary to clean up the tcpdump container that was created.

Cri-o

Cri-o uses nsenter to specify the process of the target container to enter the target network namespace and execute the tcpdump command. Since it doesn’t use a tcpdump image, the tcpdump executable must exist on the target node:

This method doesn’t create a container within the privileged pod, so there’s no need to clean up the environment.

Docker

The method for docker is similar to containerd; a tcpdump container is started and shares the network namespace with the target container:

Environment Cleanup

Since a privileged pod is created in privileged mode, and containerd and docker also create tcpdump containers within the privileged pod, it’s necessary to clean up the created tcpdump containers first, and then the privileged pod:

Conclusion

The implementation of unprivileged mode is relatively simple; there’s no need to consider container runtime issues. However, it has a drawback: the target container’s runtime environment, such as 32-bit/64-bit, amd/arm, needs to be considered, potentially requiring multiple sets of tcpdump locally to accommodate different container environments.

The implementation of privileged mode is comparatively complex. If there are other runtimes, ksniff needs to be extended for additional functionality. Also, some cluster nodes might disable privileged pods, making this method infeasible.

Despite some usage limitations, the approach to file uploads and handling different container runtimes in this article is still worth referencing.