How to Fix Server Login Error and Address Mining Virus Threats

While attempting to log into the server, I experienced a password error and couldn’t access it. The keyword is “server login error.”

I intended to reset the password on the cloud management platform but accidentally noticed a security alert, indicating the presence of a mining virus.

The mining viruses are quite invasive; they generally terminate processes that consume more than 20% of CPU usage to ensure enough resources for mining, which often results in the disruption of normal business operations.

Changing the login password is typically done to prevent competitors (other mining viruses) from brute-forcing their way in to seize computing resources, leading to the normal operation and maintenance work being affected, which is the situation discussed in this article.

Resetting the password required a server restart, making the first crime scene disappear.

It proved that the first crime scene indeed disappeared; this virus was not resilient and collapsed after a single restart.

Overall Investigation Approach

Malicious Program Investigation

1. Network Check

Using the netstat -tunlap command revealed no network anomalies, nor any expected outbound mining pool activities or abnormal process names.

2. Process Check

01. Network Process Check

No network anomalies were detected, so there was no need for network-related process checks.

02. Alert Process Check

Using ps -ef | grep to check the process IDs and startup commands mentioned in the alert details, no abnormal processes were found.

03. All Processes Check

Using the pstree -as command to view the names of all processes and the CMD of the startup processes found no abnormal processes and no expected names of random characters or processes similar to system processes.

04. Hidden Process Check

Using ps -ef | awk ‘{print}’ | sort | uniq > 1 && ps -ef | awk ‘{print}’ | sort | uniq > 2 && diff 1 2 to check for hidden processes, no abnormal processes were found.

05. Process Resource Check

Using the top command to view resource consumption by all processes detected no abnormal processes and no processes with expected high CPU or MEM usage.

3. File Check

01. Alert File Check

Using the stat command to verify malicious files mentioned in the alert details did not reveal any abnormal files.

02. Timestamp File Check

Using find / -newerct to examine the alert-mentioned attack timestamps uncovered a large number of abnormal files.

001. /tmp/2mOsaG3ceM was created by the attacker on 2023-11-30 08:24:58; file sandbox detection revealed it as a malicious program, needing subsequent backup and deletion.

002. /etc/crontab was altered by the attacker on 2023-11-30 08:25:01, adding a scheduled task to execute a malicious program. It needs to be backed up, and both the scheduled task /etc/crontab and the malicious program /tmp/2mOsaG3ceM should be deleted afterwards.

003. /var/spool/cron/root was altered by the attacker on 2023-11-30 08:25:01, adding a scheduled task to execute a malicious program. Following backup, the task /var/spool/cron/root and malicious program /tmp/2mOsaG3ceM should be deleted.

004. /root/.bash_logout was altered by the attacker on 2023-11-30 08:25:01, embedding an auto-task to execute a malicious program upon user logout from bash. Following backup, /root/.bash_logout and the malicious program /tmp/2mOsaG3ceM should be deleted.

005. /tmp/.opass was created by the attacker on 2023-11-30 08:24:58; the filename suggests it is a password file generally used for decrypting malicious programs; it should be backed up and deleted.

006. /tmp/RgX48OS6BF was modified by the attacker on 2023-11-30 10:36:40, though it was created at 2023-11-30 08:24:56. Sandbox detection indicated it as a malicious program requiring subsequent backup and deletion.

007. /root/.ssh/authorized_keys was modified by the attacker on 2023-11-30 08:24:58. It was confirmed with the system administrator (myself) that this SSH public key was added by the attacker. After backup, /root/.ssh/authorized_keys should be deleted.

008. /etc/ssh/sshd_config was modified by the attacker on 2023-11-30 08:25:01. Comparing it with the sshd_config files of other normal servers with the same configuration revealed an added PubkeyAuthentication yes by the attacker to enable public key login. If public key login is unnecessary, it should be deleted.

03. Sensitive Directory Check

Using find /tmp ! -type d -exec ls -lctr –full-time {} \+ 2>/dev/null to examine the temporary directory found no new abnormal files.

Using find $HOME ! -type d -exec ls -lctr –full-time {} \+ 2>/dev/null to examine the home directory found no new abnormal files.

04. Privileged File Check

Using find / -perm 2000 2>/dev/null to check privileged files found no abnormal files.

Backdoor Investigation

1. Account Backdoor Check

01. SSH Accounts

Using cat /etc/passwd | grep -v nologin$ to check SSH accounts that can log in found only root’s /bin/bash can log in, with no abnormal accounts.

Using cat /etc/passwd | awk -F: ‘3==0 {print 1}’ to check UID 0 superuser accounts found only root, with no abnormal accounts.

Using cat /etc/shadow | awk -F: ‘length(2)>2 {print

Checking /etc/shadow’s ctime discovered it matches the cloud management platform’s SSH password reset and server restart time. Checking /etc/passwd revealed polkitd and chrony users cannot log in. Thus concluding polkitd and chrony accounts’ SSH credentials were platform-configured, unrelated to the attack activities.

Using cat /etc/shadow | awk -F: ‘length(2)==0 {print 1,

02. SSH Keys

The /root/.ssh/authorized_keys file has been analyzed under “Malicious Program Investigation – File Check,” and will not be reanalyzed here.

2. Conditional Backdoor Check

01. Scheduled Tasks

Using find /etc/*cron* ! -type d -exec ls -lctr –full-time {} \+ 2>/dev/null to examine scheduled tasks found /etc/crontab was altered by the attacker on 2023-11-30 08:25:01. Since analysis was already performed under “Malicious Program Investigation – File Check,” it will not be reexamined here.

Using find /var/spool/cron -type f -exec ls -lctr –full-time {} \+ 2>/dev/null to examine scheduled tasks found /var/spool/cron/root was altered by the attacker on 2023-11-30 08:25:01. Since analysis was already performed under “Malicious Program Investigation – File Check,” it will not be reexamined here.

02. Startup Items

Using find /etc/rc.d/ -type f -exec ls -lctr –full-time {} \+ 2>/dev/null to examine startup items found ctime to be 2019-07-11, with no anomalies detected.

03. Autostart Services

Using chkconfig –list and service –status-all to examine autostart services found no anomalies.

04. Command Alias

Using the alias command to examine command aliases found no anomalies.

Using find / -name *bashrc* -type f -exec ls -lctr –full-time {} \+ 2>/dev/null to examine command aliases found ctime to be 2019-07-11, with no anomalies detected.

Source Investigation

1. Log Analysis

01. System Logs

001. Security Log secure

Using cat /var/log/secure* | grep Accepted | awk ‘{print $11}’ | sort | uniq -c | sort -nr revealed a malicious IP address 8.219.176.16 had successfully logged into SSH once.

Using cat /var/log/secure* | grep 8.219.176.16 | grep Failed | wc -l showed the malicious IP address 8.219.176.16 did not have any failed login attempts.

Using cat /var/log/secure* | grep 8.219.176.16 to check the malicious IP login time showed it was on 2023-11-30 08:24:40.

002. Login Log last Series

Using lastlog (/var/log/lastlog) to view user last login information and w (/var/run/utmp) to view information on users currently logged in showed no anomalies.

003. Command Log history

Using the history (/root/.bash_history) command to view historical commands showed no commands used by the attacker.

02. Middleware Logs

No middleware service is running, hence not applicable for middleware log checking.

03. Database Logs

No database service is running, hence not applicable for database log checking.

04. Security Device Logs

No security devices have been deployed, hence not applicable for security product log checking.

2. Traffic Analysis

No traffic collection product has been deployed, hence not applicable for traffic inspection.

3. Memory Analysis

The server was rebooted when the SSH password was reset, causing memory loss, so it has no analysis value.

4. Source Investigation Conclusions

01. 2023-11-30 08:24:40

The attack source IP address 8.219.176.16 (identified as a malicious IP by threat intelligence, located in Singapore, Alibaba Cloud) logged into the SSH service’s root account at 2023-11-30 08:24:40, with no failed login records, just one successful login record.

02. 2023-11-30 08:24:58

Eighteen seconds later, between 2023-11-30 08:24:58 to 08:25:01, the attacker uploaded a malicious program /tmp/2mOsaG3ceM, adding a scheduled task for this program in /var/spool/cron/root and /etc/crontab, and an additional conditional task in /root/.bash_logout.

At the same time, they created an SSH key in /root/.ssh/authorized_keys and modified /etc/ssh/sshd_config to allow key-based login.

Meanwhile, they uploaded a password file /tmp/.opass and a malicious program /tmp/RgX48OS6BF (VirusTotal indicates it requires a password to open).

03. 2023-11-30 08:34:25

Ten minutes later, the mining program executed by the attacker was detected by Alibaba Cloud’s security system at 2023-11-30 08:34:25.

04. 2023-12-01 09:34:24

A day later, the server was rebooted when resetting the SSH password, causing the malicious process to terminate and not restart.

Tool Check

01. chkrootkit

1. wget ftp://ftp.chkrootkit.org/pub/seg/pac/chkrootkit.tar.gz

2. tar -zxvf chkrootkit.tar.gz

3. cd chkrootkit-<version>

4. ./chkrootkit > result.txt

5. cat result.txt | grep -v “nothing found\|not infected\|not found\|not tested”

A new abnormal file /usr/lib/debug/usr/.dwz was discovered, but its ctime is 2019-07-11, and the file does not actually exist.

02. rkhunter

1. yum -y install rkhunter

Or

1. Download via browser https://sourceforge.net/projects/rkhunter/files/latest/download

2. tar -zxvf rkhunter-<version>.tar.gz;

3. cd rkhunter-<version>

4. ./installer.sh –install

Then

1. rkhunter -c –sk

2. cat /var/log/rkhunter/rkhunter.log | grep -C 2 Warning

Every Warning was examined, but no anomalies were found.

03. GScan

1. git clone https://github.com/grayddq/GScan.git

2. cd GScan

3. python GScan.py

The tool discovered the attacker uploaded the SSH key, which had already been addressed.

Follow-up Tasks

Serial Number	Task	Reason	Method
1	After backup, delete the files: /tmp/2mOsaG3ceM, /tmp/RgX48OS6BF, /tmp/.opass.	These are malicious programs	scp [email protected]:/tmp/2mOsaG3ceM C:\Users\onemorethink\Desktoprm -rf /tmp/2mOsaG3ceM scp [email protected]:/tmp/RgX48OS6BF C:\Users\onemorethink\Desktoprm -rf /tmp/RgX48OS6BF scp [email protected]:/tmp/.opass C:\Users\onemorethink\Desktoprm -rf /tmp/.opass
2	After backup, delete the files: /etc/crontab, /var/spool/cron/root, /root/.bash_logout.	Scheduled and conditional tasks triggering malicious programs.	scp [email protected]:/etc/crontab C:\Users\onemorethink\Desktoprm -rf /etc/crontab scp [email protected]:/var/spool/cron/root C:\Users\onemorethink\Desktoprm -rf /var/spool/cron/root scp [email protected]:/root/.bash_logout C:\Users\onemorethink\Desktoprm -rf /root/.bash_logout
3	After backup, delete the file: /root/.ssh/authorized_keys.	The attacker added an SSH key backdoor.	scp [email protected]:/root/.ssh/authorized_keys C:\Users\onemorethink\Desktoprm -rf /root/.ssh/authorized_keys
4	Modify the configuration file to prohibit SSH key login: /etc/ssh/sshd_config.	This was a configuration enabled by the attacker for public key login.	sed -i ‘s/PubkeyAuthentication yes/PubkeyAuthentication no/g’ /etc/ssh/sshd_config
5	Change the password for the root account of the SSH service	The root account was successfully logged in by the attacker.	passwd root

When modifying the configuration file to prohibit SSH key login, it was found that /etc/ssh/sshd_config could not be modified because it had been assigned the i attribute. The i attribute must be removed before modification is possible.

Further investigation showed that other files slated for deletion also had parts assigned the i attribute, requiring removal of the i attribute before deletion can proceed.

Other Notes

1. If you wish to practice in this environment later, executing the command within the scheduled task on your Linux environment will suffice. It is recommended to use a Host-Only mode virtual machine to avoid worm proliferation; if the situation becomes uncontrollable, you can execute the following command to terminate all malicious processes, or disconnect the network or power off.

2. There is a point of suspicion: although the malicious program’s scheduled task runs every minute, it is not effective in practice, and restarting crond immediately doesn’t help. However, if the commands in the scheduled task are executed manually, they execute successfully.

EventBackupLoginProcessLog