Background Summary
The host security program on a certain machine crashed abnormally, and the process repeatedly failed to start. On-site investigation found that the host had an OOM memory overflow, speculating that the host security agent used more memory than the limit and was killed.
However, the system itself had enough memory, with 14GB remaining.
Investigation Approach
Check Kernel Log Information with dmesg
dmesg --level=err,warn -T
Sometimes we only need error and warning logs, not information of other levels. Usually, OOM logs are at the error level.
dmesg | grep oom
Abnormal Positioning
The host security alert task file /var/spool/cron/dump.rdb was tampered with multiple times.
Investigation found that there was a dump.rdb file in the /var/spool/cron directory, which was over 600MB in size and still being written to.
The dump.rdb file is a Redis backup file, which should not normally appear in the cron job directory. Writing files to the cron job directory is the most common way for Redis unauthorized access intrusions.
Based on the creation time of dump.rdb being a few months ago and the lack of logs in Redis, it cannot be confirmed, but it is speculated that the Redis on this host had a vulnerability and was previously compromised.
Cron Job Logs
By checking the scheduled task logs, it can be determined that the earliest execution of the dump.rdb scheduled task was on 2022-07-13 10:12:01. It can be seen in the cron logs that the task did not execute successfully because the dump.rdb user did not exist in /etc/passwd, causing no actual impact.
Testing shows that the filenames of scheduled tasks in the /var/spool/cron/ directory must be usernames existing in /etc/passwd to be executed. However, in /etc/rc.d/, any filename can be used.
Linux Scheduled Tasks
Investigation Conclusion
The large size of the /var/spool/cron/dump.rdb file caused the host security agent to read this file when scanning scheduled tasks, exceeding the maximum memory limit and causing the system to kill the agent due to OOM.
To ensure that the host security program does not excessively occupy system resources, it typically limits its own CPU and memory usage.
Under normal circumstances, there should not be files of several hundred megabytes in the cron job directory. During the scanning process, the agent reads the entire dump.rdb file into memory at once, exceeding the memory limit, and the agent is killed by the system.
Agent Optimization Ideas
This case highlights a scenario that was not considered.
- The agent should check the file size before reading it.
- Do not read the entire file into memory at once.