Confluence suddenly stops responding and now we cant start it up.

[Johan Johansson]

Hi all, 

we have successfully ran Jira and Confluence on a Debian distribution in Google cloud for a number of years. We rarely update the versions.

During a moment when (from what we know) no one was using Confluence, the CPU started to run at 100%.

I have rebooted server to no avail.

Starting up with sudo ./start-confluence.sh gives no error, and nothing (as in, not a single line) is written to the atlassian-confluence.log file.

Running "top -c" gives this list:

18289 conflue+ 20 0 253036 6380 0 S 197.1 0.1 70:25.46 /boot/vmlinuz
1503 conflue+ 20 0 13280 3068 2800 S 0.0 0.0 0:00.72 ./o3LPbfh ./nafOeFd
1504 conflue+ 20 0 13292 3036 2756 S 0.0 0.0 0:00.86 ./o3LPbfh ./0J6BPhj
1505 conflue+ 20 0 13820 3540 2732 S 0.0 0.0 0:06.78 ./o3LPbfh ./prot
30982 conflue+ 20 0 5808 704 620 S 0.0 0.0 0:00.00 sleep 10
30991 conflue+ 20 0 5808 704 620 S 0.0 0.0 0:00.00 sleep 5
30995 conflue+ 20 0 5808 700 620 S 0.0 0.0 0:00.00 sleep 1

So something is happening, but is unclear what.

The mysql database that is used as the storage is responsive and looks ok from what I can see.

The disk is not full.

A local "wget http://localhost:8090" gives "connection refused"

JIRA is running fine.

My limited Linux skills has now reached the end of the line so I am asking for assistance. I do have the possibility to bring back a 5 day old backup but as we use Jira on the same server I first need to understand what the reason is...

Uname output:

Linux support 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux

Confluence version: 6.2.1

I have tried to start in safe mode, with the same result.

I tried starting up with the "-fg" parameters and the following came at the end:

15-Apr-2019 15:59:47.266 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.43
Killed

Anyone please?

[Daniel Eads · Accepted solution]

Hi Johan,

Based on your Confluence version, symptoms, and output from top showing the suspicious looking processes (./o3LPbfh ./nafOeFd in your output) it sounds like your instance might be affected by an opportunistic attack against the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20). We've seen an infection going around that injects malware and the bitcoin miner it tries to run uses all the CPU available on the box. Initially the kerberods malware was being deployed as the payload, but other attacks might be trying to inject different payloads.

I'd recommend tackling things in this order:

1. Kill malicious processes
2. Clean up your crontab
3. Upgrade Confluence
4. Use a malware scanner to find remaining malware traces

Malicious processes

The top command you ran shows several malicious processes running under the confluence user. If Confluence is currently stopped, you can probably plan on killing all processes running as the confluence user. Note the process ID (pid) from the top output and then kill the process using kill -9 followed by the pid. Example:

sudo kill -9 1505

Clean up your crontab

Since most malware adds a cronjob that relaunches the malware every few minutes, you'll also need to check the crontab file and remove any suspicious-looking entries. For Ubuntu, this is stored in the /var/spool/cron/crontabs/ directory. Normally you should use the crontab command to edit the crontab, but for cleanup purposes we'll be inspecting the file for any pre-existing entries.

Using vim (or whichever text editor you're comfortable with), you'll open the file and remove suspicious-looking jobs.

sudo vim /var/spool/cron/crontabs/confluence

Confluence comes up on system startup through the SysV/systemd daemons, so we would expect the confluence user's crontab to not exist under normal circumstances. It's most likely the case that any entries in this file are malicious, but make sure you check them before deleting them entirely.

Upgrade Confluence

Once your CPU is under control and new malicious process aren't spawning, you need to upgrade Confluence to a version that isn't affected by the vulnerability. I'd recommend looking at one of these versions (latest releases as of this post):

• 6.6.13 (6.6 is an Enterprise release)
• 6.13.4 (6.13 is an Enterprise release)
• 6.14.3
• 6.15.2

Use a malware scanner

Finally, you need to clean up any remaining traces of malware on your system. The LSD malware cleanup tool will be useful for removing the Kerberods malware. Other malware payloads might need different cleanup tools depending on which attack and payload were used. A good starting place for detecting other types of infections are the scanners linked here. Once a particular infection is identified, googling for "____ removal tool" is a good place to start if the scanner was unable to remove the malware automatically.

Please let me know if you have more questions!
Daniel | Atlassian Support

[Johan Johansson]

Yes thanks this was exactly it.