Comprehensive Guide to Identifying and Mitigating Malicious Processes on a Compromised Server

If the alarm file cannot be located at this stage, it may be deleted by the malicious program itself. At this time, other landing files can be checked based on the alarm time. If there is still no malware, it may be necessary to perform a memory check and kill.

Cross-verify using Weibu sandbox; the conclusion of the dhpcd analysis report [1] is that it is a malicious sample. Also, this was the first submission, indicating we have led the way in identifying the mining virus issue.

02. Process

Use the command lsof /usr/bin/dhpcd to find that this malicious file is being used by a process with ID 14213.

Use the command pstree -asp 14213 to find all child processes of this process, all of which need to be terminated subsequently.

03. Network

Use the command netstat -tunlap | grep 14213 to find that the malicious process has external connections to port 3389 of 93.240.52.232 and port 443 of 139.99.125.38, suspected as a remote control server and a mining pool server respectively, based on threat intelligence.

Use the command tcpdump -i eth0 $host 93.240.52.232 and port 3389$ or $host 139.99.125.38 and port 443$ -w dhpcd.pcap to capture traffic packets and download them for local analysis.

Analysis reveals that the remote server 93.240.52.232:3389 issues mining tasks for `algo coins` to the victim server every 10 seconds. However, since mining pool 139.99.124.170:443 is unresponsive, it’s unclear if the victim server was able to perform mining tasks, hence the victim server did not actually begin mining.

Moving forward, requests to malicious IP addresses 93.240.52.232 and 139.99.125.38 need to be blocked at the edge firewall, and recent 7-day traffic records should be checked on network security devices to ascertain similar requests; if such requests are present, assess whether the source device is also infected by the mining virus.

1.1.2. Time File Investigation

Use the command find / -newerct '2024-01-24 08:10:00' ! -newerct '2024-01-24 09:10:00' ! -path '/proc/*' ! -path /'sys/*' ! -path '/run/*' -type f -exec ls -lctr --full-time {} \+ 2>/dev/null to investigate files deposited before and after the malicious program landed, finding a few suspicious files.

01. /etc/shadow

No backdoor accounts in /etc/shadow; attacker enhanced security by changing the root’s weak password to protect their malicious process from mining and ransomware viruses. Root SSH password change recommended.

02. /root/.ssh/authorized_keys

This is an SSH backdoor public key file created by the attacker for logging into the SSH service; it needs to be backed up and then deleted.

03. /usr/bin/dhpcd

It’s the malicious program identified in 1.1.1 Alert File Investigation.

04. /etc/rc.d/rc.local

The attacker created a startup service to execute the malicious program /bin/dhpcd, which needs to be removed along with the malicious program.

05. /var/spool/cron/root

The attacker has set up a scheduled task that runs the malicious program /bin/dhpcd every 7 minutes past the hour, which needs to be deleted along with the malicious program.

06. Suspicious but Normal Files

The ctime of /etc/crontab is close to the landing time of the malicious program /bin/dhpcd, but the file itself does not contain any abnormal content, thus it is considered normal.

1.1.3. Sensitive Directory Investigation

Use the command find /tmp ! -type d -exec ls -lctr --full-time {} \+ 2>/dev/null to investigate temporary directories; no new suspicious files were found.

Use the command find $HOME ! -type d -exec ls -lctr --full-time {} \+ 2>/dev/null to investigate home directories; no new suspicious files were found.

1.1.4. Privileged File Investigation

Use the command find / -perm -u=s 2 -type f -ls >/dev/null to investigate privileged files; no suspicious files were found.

1.2. Network Investigation

Use the command netstat -tunlap and besides discovering the dhpcd process analyzed in 1.1.1 Alert File Investigation, an additional connection on port 22 with IP 115.79.35.110 was found. Weibu confirms this is a malicious IP, but after examining cat /var/log/secure* | grep 115.79.35.110, it’s identified as a brute force attempt without successful login, thus can be ignored.

1.3. Process Investigation

1.3.1. Network Process Investigation

No new malicious processes were found during the network investigation, so this step is skipped.

1.3.2. Comprehensive Process Investigation

Use the command pstree -as to investigate all processes by name and startup command; no other suspicious processes were found.

1.3.3. Hidden Process Investigation

Use the commands ps -ef | awk '{print}' | sort | uniq > 1, ps -ef | awk '{print}' | sort | uniq > 2, diff 1 2 to investigate hidden processes; apart from the known malicious process /bin/dhpcd and the brute-forced SSH service, there were no new findings.

1.3.4. Process Resource Investigation

Use the command top to review process resource consumption; apart from the well-known malicious process dhpcd maxing out CPU to 200%, there were no further discoveries.

2. Backdoor Investigation

2.1. Account Backdoor Investigation

2.1.1. SSH Accounts

Using the command cat /etc/passwd | grep -v 'nologin\|false' to check SSH login accounts, it was found that only root and lighthouse accounts have /bin/bash logged-in, and no abnormalities were found.

Using the command cat /etc/passwd | awk -F':' '$3==0 {print $1}' to check for superuser accounts with UID 0, only the root account was found, with no abnormalities.

Using the command cat /etc/shadow | awk -F':' 'length($2)==0 {print $1}' to search for SSH accounts with empty passwords, no anomalies were found.

2.1.2. SSH Keys

The privileged user’s /root/.ssh/authorized_keys file has been analyzed in 1.1.2.2; ordinary user accounts showed no SSH key files.

2.2. Conditional Backdoor Investigation

2.2.1. Scheduled Tasks

Using the command find /var/spool/cron/ -type f -exec ls -lctr --full-time {} \+ 2>/dev/null to investigate scheduled tasks, it’s found that the /var/spool/cron/root file was modified by the attacker on 2024-01-24 08:14:16, already analyzed in 1.1.2.5.

Using the command find /etc/*cron* -type f -exec ls -lctr --full-time {} \+ 2>/dev/null to investigate scheduled tasks, it was found that the /etc/crontab file was altered on 2024-01-24 08:11:09, but showed no anomalies, analyzed in 1.1.2.9.

2.2.2. Startup Items

Using the command find /etc/rc.d/ -type f -exec ls -lctr --full-time {} \+ 2>/dev/null to inspect startup items found the /etc/rc.d/rc.local file was modified by the attacker on 2024-01-24 08:14:08, already analyzed in 1.1.2.4.

2.2.3. Autostart Services

Using commands chkconfig --list and service --status-all to inspect autostart services, with no anomalies detected.

2.2.4. Command Aliases

Using the alias command to inspect command aliases, no anomalies were found.

Using the command find / -name *bashrc* -type f -exec ls -lctr --full-time {} \+ 2>/dev/null to inspect configuration files allowing command aliases, no anomalies were found.

3. Trace Investigation

3.1. Log Analysis

3.1.1. System Logs

01. Security Log secure

Using the command cat /var/log/secure* | grep Accepted | awk '{print $11}' | sort | uniq -c | sort -nr to discover five unknown malicious IP addresses successfully logged into SSH once each.

Weibu confirms 222.186.16.196 as a malicious IP in Beijing. Using the command cat /var/log/secure* | grep 222.186.16.196 | grep Failed | wc -l revealed 4 failed login attempts, and with the command cat /var/log/secure* | grep 222.186.16.196 | grep Accepted | wc -l, 1 successful login attempt was identified, indicating a brute-force success.

Using the command cat /var/log/secure* | grep 222.186.16.196 to view the login time, the brute-force attack began at 2024-01-24 04:07:11, succeeding 6 seconds later, indicating it might not be related to this mining event.

Weibu confirms 222.186.16.186 as a malicious IP in Beijing. Using the command cat /var/log/secure* | grep 222.186.16.186 | grep Failed | wc -l found 4 failed login records, and using cat /var/log/secure* | grep 222.186.16.186 | grep Accepted | wc -l found 1 success, indicating another brute-force success.

Viewing login times with cat /var/log/secure* | grep 222.186.16.186, it began at 2024-01-24 05:48:58 with success 6 seconds later, yet again unrelated timing with this mining event.

Observation using cat /var/log/secure* | grep 222.103.98.36 shows a successful login at 2024-01-24 08:10:27, correlating directly with the mining event’s start, marking them as a key perpetrator.

02. Login Logs last series

Using the command lastlog to check the last login details (located at /var/log/lastlog) and the command w to check currently logged-in users (using /var/run/utmp), no anomalies were identified.

03. Command Logs history

By reviewing the history with the history command (/root/.bash_history), no commands used by the attacker were found.

3.1.2. Middleware Logs

No running middleware services, no middleware log investigation performed.

3.1.3. Database Logs

No database services running, hence database logs are not analyzed.

3.1.4. Security Device Logs

No security devices deployed, hence security product logs are not analyzed.

3.2. Traffic Analysis

No traffic collection products deployed, hence traffic investigation is not conducted.

3.3. Memory Analysis

The server was rebooted when resetting the SSH password, making memory unavailable for analysis.

3.4. Investigation Conclusion

01. 2024-01-24 08:10:27

The attacker from South Korea with source IP address 222.103.98.36 logged into the SSH service with the root account; only one successful login with no failures.

02. 2024-01-24 08:11:09 to 08:14:16

One minute later, the attacker uploaded and executed the malicious program /usr/bin/dhpcd, modified the backdoor configuration /etc/shadow, /root/.ssh/authorized_keys, /etc/rc.d/rc.local, and /var/spool/cron/root, all within a span of 3 minutes.

Notably, the modification time of the /etc/crontab file also falls within the attacker’s activity window, although no anomalies were found.

03. 2024-01-24 08:13:21

The malicious program /usr/bin/dhpcd was uploaded by the attacker at 2024-01-24 08:13:19, and by 08:13:21, Tencent Host Security completed threat detection with alert notification, showcasing impressive capabilities.

4. Pending Actions

4.1. Terminate Malicious Processes

Serial Number	Action Item	Reason	Method
1	Terminate malicious process dhpcd	Linked to a remote control server 93.240.52.232:3389 for mining algo coin tasks, then connects to a mining pool server 139.99.124.170:443 to submit mining results.	kill -9 14213 kill -9 14214 kill -9 14215 kill -9 14216 kill -9 14217 kill -9 14218 kill -9 14855 kill -9 14856

4.2. Backup and Delete Malicious Program

Serial Number	Action Item	Reason	Method
1	Backup and delete /usr/bin/dhpcd	Linked to a remote control server 93.240.52.232:3389 to get algo coin mining tasks, then connects to a mining pool server 139.99.124.170:443 to submit results.	scp [email protected]:/usr/bin/dhpcd ./ rm -rf /usr/bin/dhpcd

4.3. Repair Vulnerabilities and Backdoors

Serial Number	Action Item	Reason	Method
1	Change root user password	Weak password exploit allowed attacker to gain server access	passwd root
2	Backup and delete /root/.ssh/authorized_keys	Attacker’s SSH backdoor key	scp [email protected]:/root/.ssh/authorized_keys ./ rm -rf /root/.ssh/authorized_keys
3	Backup and modify /var/spool/cron/root	Attacker’s backdoor scheduled task	1. Open file: vim /var/spool/cron/root 2. Navigate: using arrow keys to malicious code 3. Remove code: Enter `dd` to delete malicious code 4. Exit file: :wq
4	Backup and modify /etc/rc.d/rc.local	Attacker’s backdoor startup items	Same as above.

4.4. Attack Traceback and Protection

Serial Number	Action Item	Reason	Method
1	Protection: Block requests to malicious IP addresses 93.240.52.232 and 139.99.125.38 at the boundary firewall.	93.240.52.232 is the remote control server, 139.99.125.38 is the mining pool server.	Refer to the boundary firewall’s operational manual.
2	Traceback: Search network security devices for the last 7 days to see if there are related access requests. If found, the source device should be inspected for mining virus infection.	As above.	Refer to the network security device’s operational manual.

Sax2 Intrusion Prevention System