In daily emergency response, malicious dnslog requests initiated by Linux hosts often appear, but the corresponding triggering process cannot be found. There is no good monitoring method in user space, so systemtap is used here to monitor through the kernel.
What is SystemTap
Systemtap is a tool that uses the API provided by Kprobe to dynamically monitor and trace the running Linux kernel. Compared to Kprobe, systemtap is simpler, providing users with a simple command-line interface and a scripting language for writing kernel instructions, similar to C and D languages.
SystemTap Principles
The basic idea of SystemTap is to name events and provide handlers for them. Whenever the specified event occurs, the kernel treats the handler as a subroutine, runs it, and then continues running. There is a series of events, such as entering or exiting a function, timer expiration, or the start and stop of the entire SystemTap session. The handler is a series of scripting language statements that specify the work to be done when the event occurs. This work usually involves extracting data from the event context, storing it in internal variables, or printing the results.
SystemTap works by translating scripts into C, executing a C compiler to create a kernel module. When the module is loaded, it activates all probe events by mounting to the kernel. Then, when an event occurs on any processor, the compiled handler runs. Finally, the SystemTap session stops, the hooks are removed, and the kernel module is unloaded. The entire process is driven by the command-line program stap.
The principle is shown in the figure below:
SystemTap Installation
Using an older system (CentOS6.8 x64 as an example), download the debug dependency package that exactly matches the kernel version. Systemtap mainly relies on the following three kernel modules.
- kernel-debuginfo-common
- kernel-debuginfo
- kernel-devel
Check the current system kernel version
[root@VM-0-4-centos ~]# uname -r 2.6.32-642.6.2.el6.x86_64
Install systemtap
yum install systemtap systemtap-runtime -y
Check the dependent kernel information package
[root@VM-0-4-centos ~]# stap-prep Need to install the following packages: kernel-devel-2.6.32-642.6.2.el6.x86_64 kernel-debuginfo-2.6.32-642.6.2.el6.x86_64 Loaded plugins: fastestmirror, security Setting up Install Process Loading mirror speeds from cached hostfile No package kernel-devel-2.6.32-642.6.2.el6.x86_64 available. No package kernel-debuginfo-2.6.32-642.6.2.el6.x86_64 available. Error: Nothing to do package kernel-devel-2.6.32-642.6.2.el6.x86_64 is not installed package kernel-debuginfo-2.6.32-642.6.2.el6.x86_64 is not installed problem installing rpm(s) kernel-devel-2.6.32-642.6.2.el6.x86_64 kernel-debuginfo-2.6.32-642.6.2.el6.x86_64
stap-prep will print the dependent packages and their versions. Note: Do not install through yum, yum will not match the corresponding kernel version, even if installed, it cannot be used.
Google search for the corresponding dependency version
Note: systemtap has strict requirements for kernel versions, and you must download and install the fully matching version.
Download the corresponding packages in sequence.
Install the corresponding kernel information package with rpm
[root@VM-0-4-centos ~]# rpm -ivh kernel-debuginfo-2.6.32-642.6.2.el6.x86_64.rpm
error: Failed dependencies:
kernel-debuginfo-common-x86_64 = 2.6.32-642.6.2.el6 is needed by kernel-debuginfo-2.6.32-642.6.2.el6.x86_64
The error indicates that the installation of kernel-debuginfo depends on kernel-debuginfo-common, so you need to download and install kernel-debuginfo-common first. Similarly, find the corresponding package through Google.
[root@VM-0-4-centos ~]# rpm -ivh kernel-debuginfo-common-x86_64-2.6.32-642.6.2.el6.x86_64.rpm
warning: kernel-debuginfo-common-x86_64-2.6.32-642.6.2.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Preparing... # [100%]
1:kernel-debuginfo-common# [100%]
[root@VM-0-4-centos ~]# rpm -ivh kernel-debuginfo-2.6.32-642.6.2.el6.x86_64.rpm
Preparing... # [100%]
1:kernel-debuginfo # [100%]
[root@VM-0-4-centos ~]# rpm -ivh kernel-devel-2.6.32-642.6.2.el6.x86_64.rpm
Preparing... # [100%]
package kernel-devel-2.6.32-754.35.1.el6.x86_64 (which is newer than kernel-devel-2.6.32-642.6.2.el6.x86_64) is already installed
Note the pitfall: When we previously installed systemtap through yum, the automatically installed kernel-devel did not match the kernel version. Here you need to uninstall the automatically installed kernel-devel.
[root@VM-0-4-centos ~]# yum remove -y kernel-devel-2.6.32-754.35.1.el6.x86_64 [root@VM-0-4-centos ~]# rpm -ivh kernel-devel-2.6.32-642.6.2.el6.x86_64.rpm Preparing... # [100%] 1:kernel-devel # [100%]
Check if systemtap is installed
[root@VM-0-4-centos ~]# stap-prep
If there is no output when executing stap-prep, it means that all dependencies are installed, and systemtap can be used normally. However, uninstalling kernel-devel will cause problems with systemtap itself, so you need to reinstall it with yum.
yum install systemtap systemtap-runtime -y
systemtap Testing
[root@VM-0-4-centos ~]# stap -e 'probe begin{printf("Hello, World\n"); exit();}'
Hello, World
[root@VM-0-4-centos ~]# stap -ve 'probe begin{printf("Hello, World\n"); exit();}'
Pass 1: parsed user script and 111 library script(s) using 206796virt/34536res/3220shr/31672data kb, in 80usr/10sys/97real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 207456virt/35524res/3524shr/32332data kb, in 10usr/0sys/3real ms.
Pass 3: using cached /root/.systemtap/cache/3e/stap_3e793d5afad1dc01d547399a819b8745_979.c
Pass 4: using cached /root/.systemtap/cache/3e/stap_3e793d5afad1dc01d547399a819b8745_979.ko
Pass 5: starting run.
Hello, World
Pass 5: run completed in 0usr/0sys/327real ms.
You can see that systemtap printed the string, and the -v parameter showed the execution process of the stap script. Viewing the converted C code, it can be seen that systemtap first converts the stap script to C, then compiles the C into a ko module and loads it into the kernel for execution.
Workflow Explanation
Important Files and Directories
| /lib/modules/KERNEL_VERSION/systemtap/ | Stores SystemTap tool modules. |
| /usr/share/systemtap/tapset/ | Stores standard tapset libraries. |
| /usr/share/doc/packages/systemtap/examples | Contains multiple example SystemTap scripts for various purposes. Available only if the systemtap-docs package is installed. |
| ~/.systemtap/cache | Data directory for cached SystemTap files. |
| /tmp/stap* | Temporary directory for SystemTap files, containing translated C code and kernel objects. |
The default file extension for SystemTap scripts is .stp, and the file content follows a fixed format with complete syntax (variables, constants, comments, conditions, loops, arrays, etc.). begin/end mark the start and end of a SystemTap session, respectively. The recommended IDE is VSCode. As shown below:
Scripts
SystemTap script writing reference: https://sourceware.org/systemtap/tutorial/
Probes
Kprobes allow you to install pre-handlers and post-handlers for any kernel instruction as well as function entry and return handlers.
Commonly used probe events are as follows:
| tid() | ID of the current thread. |
| pid() | Process ID of the current thread. |
| uid() | ID of the current user. |
| cpu() | Current CPU number. |
| execname() | Name of the current process. |
| gettimeofday_s() | Seconds elapsed since the Unix epoch (January 1, 1970). |
| ctime() | Converts time to a string. |
| pp() | String describing the current probe point being handled. |
Function reference: https://sourceware.org/systemtap/tapsets/index.html
SystemTap Applications
Monitoring DNS Requests
Script principle: Determine if the target port for outgoing connections is 53.
#! /usr/bin/env stap
global the_dport = 53
probe netfilter.ip.local_out {
if (the_dport == dport)
printf("%s[PID:%d,TID:%d] sent packet to %s:%d\n", execname(), pid(), tid(), daddr, dport)
}
Result:
Monitoring ICMP Requests
Script principle: netfilter.ip.local_out is for all outgoing protocols, with ICMP’s target port being 0.
#! /usr/bin/env stap
probe netfilter.ip.local_out {
if (0 == dport)
printf("%s[PID:%d,TID:%d] sent %d to %s:%d\n", execname(), pid(), tid(),length, daddr, dport)
}
probe netfilter.ip.local_in {
if (0 == sport)
printf("%s recv %d from %s:%d\n", execname(),length, saddr, sport)
}
Result of monitoring ping:
Conclusion
SystemTap is very powerful and has many applications beyond what is covered here. However, the installation process for SystemTap is quite cumbersome and not very user-friendly. Future considerations may include using other methods (eBPF) as a replacement.