Confluence and Java Maxing CPU

[Jack]

Ubuntu Server: v20.04.3 LTS

Confluence Server: v7.14.2

Today I updated an instance of Confluence from v7.12.0 to v7.14.2. Installation went fine and program started correctly but subsequently crashed. Looking at the processes I found several instances of /jre/bin/java slamming the processor to 100%. It looks like this is what is crashing confluence.

• Looking at the processes for user confluence, I found a process for "JavaUpdate". I killed this processes, stopped/started Confluence and the program came up as expected.
  
  
• About 20 minutes later, the Java Update process came back and crashed the program. So I again killed the process, stopped and started confluence again and it comes right back. I am now waiting to see if this happens again.

I am not sure what is forcing JavaUpdate to start or keep restarting after the process is terminated. Can anyone tell me where next to look? Is there an old program that was not removed during the update that should have been? Do I need to update one of the configuration files?

Help would be greatly appreciated.

Thank You in advance!

Jack

Update: If the server is rebooted, the problem starts all over again. I have to kill the process JavaUpate to return Confluence to its operating status.

[Andreas Schmitt]

Hello,

we have the same issue since a few days. first time it apears with confluence server 7.16.x, now we have 7.18.3 and it still the same.

problem is, my colegue trying to attach an excel document and uploads a new version with goedit and the process "openJDK Plattform binary" starting to rise up to 100%, the upload is canceled and this process still takes about 90-100% for a while.

is there any solution out there for this issue?

at the event i've got the following in the catalina*.log

26-Jul-2022 09:09:16.774 WARNUNG [Catalina-utility-3] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8090-exec-10] (id=[251]) has been active for [64.357] milliseconds (since [26.07.22 09:08]) to serve the same request for [http://192.168.0.87:8090/plugins/editor-loader/editor.action?parentPageId=3178570&pageId=7077964&spaceKey=TEC&atl_after_login_redirect=%2Fdisplay%2FTEC%2FEntwicklung%2B-%2BHeidelberg%2BEngineering&timeout=12000&_=1658819289783] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at java.base@11.0.14.1/java.net.SocketInputStream.socketRead0(Native Method)
at java.base@11.0.14.1/java.net.SocketInputStream.socketRead(Unknown Source)
at java.base@11.0.14.1/java.net.SocketInputStream.read(Unknown Source)
at java.base@11.0.14.1/java.net.SocketInputStream.read(Unknown Source)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)

..................... followed by a long, verly long list................

..................... at the end ................

 at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base@11.0.14.1/java.lang.Thread.run(Unknown Source)
26-Jul-2022 09:09:36.785 WARNUNG [Catalina-utility-3] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [http-nio-8090-exec-10] (id=[251]) was previously reported to be stuck but has completed. It was active for approximately [77.394] milliseconds.

[Daniel Ebers]

Hi @Andreas Schmitt

the other comments in this threads suggested the servers were compromised, in your case I would in first step look into the performance troubleshooting documentation rather as long as you can rule out the server's being compromised:

• Best practices for performance troubleshooting tools
  https://confluence.atlassian.com/kb/best-practices-for-performance-troubleshooting-tools-652444218.html

However, there are plenty of reasons why you are facing the trouble, a deeper look into the system would be needed.

Regards,
Daniel

[Jack]

Good morning and happy holidays,

So after digging deeper into the issue I was seeing, it is indeed the malware issue. Last weekend we deleted a suspect entry out of the crontab file and the processes stopped. However other issues were appearing. So the comments about reinstalling, unfortunately, is warranted. I started the reinstall process early in the week and with the reinstall, taking the time to install other system updates as well.

[Jack]

Update: 12:32p. PST

As I have been digging further into the compromised system this morning, I found that the PostgreSQL users and passwords may have been exposed. I would recommend changing the user name and passwords as you reinstall. Also as a precaution? I would change the passwords on the application user accounts as well.

[Jona Lama]

Hello all,

I had the same problem until today. Now instead of javaUpdate, are showing unknown proccesses. I try to kill them but they appear with another name and PID. they created folders within confluence installer folder and also they modified crontab file. 

I think its a ddos attack. How can i get rid of it?

[jpedrot]

You should reinstall your system to be 100% sure.

But remove the crontab entry and killall the processes and remove the files.

Upgrade confluence to latest version.

Monitor if something weird is going on, look at Netdata (https://www.netdata.cloud/) for easy monitoring. 

[Nic Brough -Adaptavist-]

Javaupdate suggests you've got something on the computer that is trying to upgrade Java when it detects a new release.

This will interfere with anything using the Java install it is trying to update (and the update can thrash a machine sometimes, so that might be the cause of the 100% CPU)

If this is a Windows machine, that would make sense - the standard installs of most Javas will, by default, install a small service that checks for new versions of Java and upgrades them automatically.   They can be configured to ask first, but it depends on which one it is.  (On linux boxes, it is similar, but you have to do quite a bit of config to allow it to run the upgrade without asking the humans, and it tends to complain that Confluence is running as well)

The easiest "fix" for this is to stop Confluence, upgrade to the latest Java version (after checking it is valid for Confluence 7.14.2) by triggering the upgrade, or just waiting until it kicks in on-schedule, and then starting Confluence after it finishes.

[Jack]

I am just not seeing where this process is being generated from.

Here is the process that is maxing out the CPU

F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
1 S conflue+ 6128 1 99 80 0 - 800449 ep_pol 2576132 6 22:17 ? 04:15:01 /usr/java/latest/bin/java -Djava.util.logging.config.file=/data/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx5120M -server -XX:+UseParallelGC -Djavax.net.debug=ssl -Dignore.endorsed.dirs= -classpath /data/apache-tomcat/bin/bootstrap.jar:/data/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/data/apache-tomcat -Dcatalina.home=/data/apache-tomcat -Djava.io.tmpdir=/data/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

I found the JavaUpdate file in /var/tmp/.Javadoc

The Java program being used is in /var/www/confluence/jre. It is dated 07/20/2021.

The process is saying that I have a /usr/java/latest/bin/java folder which I am unable to find.

Since the release of java is the one that is installed with the Confluence application I would have thought it would be already updated.

I am still not sure where this is coming from.

Jack

[jpedrot]

I am also seeing this on our Confluence/Jira server, I wonder if we have been hacked and something is disguising it self as javaupdate.

Using Debian 10 in our case.

Did you find a solution Jack?

[Daniel Ebers]

@jpedrot It is likely but needs further inspection.

As a general rule:

I found the JavaUpdate file in /var/tmp/.Javadoc

While not a prove of a hack, files hiding with a leading dot from users CAN be intentionally set but in such case it looks more like a hack, indeed.