khugepageds eating all of the CPU

[dovi5988]

https://community.atlassian.com/t5/Confluence-questions/How-come-my-confluence-installation-was-hacked-by-Kerberods/qaq-p/1054605

[Deleted user]

David gave me the original idea.

I can't take this system out of production atm for the updates, so I followed the mitigation steps for now (disable Widget and WebDAV plug-ins), but it's been 30 min since I did that and no more suspicious looking activity

[Deleted user]

I don't think Atlassian watches these forums...thank you for your sharing the link.

[dovi5988]

I assume it's this: https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html

[Deleted user]

Yeah, there was a malicious looking cronjob. How did that happen? Has this happened to you before?

[eleven12]

I am having the same problem on my Linux server running Ubuntu 16.04 LTS. When doing a "top" command, khugepageds is at 100% on each core. It and kerberods are both listed as having user "conflue+". I have added the line to the grub file to set transparent_hugepage=never to disable it at boot time, set "never" in the transparent_hugepages/enabled and defrag files to disabled it at run time, and run hugeadm to disable it. Nothing works. I can kill the process, but it will restart after a while. Checking the /sys/kernel/mm/transparent_hugepages/enabled files shows "always madvise [never]", and checking the number of huge_pages being managed is at 0.  I have stopped Confluence. But nothing can stop khugepageds! :)

I am suspicious that it is at least Confluence (or Java) related due to the user being Confluence. I thought 

[dovi5988]

Can someone from Atlassian reach out to me directly on this?

[Deleted user]

Same thing here...we've been running Confluence without issue now for a few years, now this "khugepageds process is at near 100% utilization" issue is popping up as of yesterday.

I can kill the process, but it starts backup a few minutes later.

I rebooted the server yesterday and it started again a few hours later.

I temporarily froze the process with (of course the number is the pid):

sudo kill -STOP 12128

...but that doesn't seem like a good solution. EDIT: After 3 hours it apparently killed the stopped process and spawned a new one as I just had to do the same thing with a new PID.

Confluence bug?

Another EDIT: This article isn't directly about Confluence... it looks like THP can be disabled at boot (https://confluence.atlassian.com/bamkb/performance-issue-with-red-hat-enterprise-linux-rhel-781189906.html -> https://access.redhat.com/solutions/46111) but I cannot test this during business hours.