confluence editor doesn't load and some time later the instance crash

[Fehér Lajos]

Hi!

We have a fine working confluence till now. We haven't changed anything in our configuration. The problem is when i want to edit a page, the editor not loaded and the page hang - but the confluence still working and able to response other requests.

After several minutes the whole instance crash.

There aren't any error messages in logs. At OS level i realized 2 process, which consume much cpu and run as confluence user.

top - 21:16:01 up 15 min, 1 user, load average: 3.60, 2.25, 1.76
Tasks: 150 total, 2 running, 148 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.8%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 15132972k total, 3631532k used, 11501440k free, 147536k buffers
Swap: 8420344k total, 0k used, 8420344k free, 1524280k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
18196 confluen 20 0 22920 5804 4 S 116.6 0.0 0:21.19 khugepageds 
18152 confluen 20 0 106m 14m 4 R 82.1 0.1 0:22.20 kerberods 
7 root 20 0 0 0 0 S 0.3 0.0 0:00.66 rcu_sched 
16254 jira2 20 0 3338m 452m 18m S 0.3 3.1 0:35.96 java 

and there are 2 files in /tmp directory

# ls -l 
összesen 3660
-rw-r--r--. 1 root root 897 ápr 17 07.48 dhclient-exit-hooks.log
drwxr-xr-x 2 confluence confluence 4096 ápr 17 06.44 hsperfdata_confluence
drwxr-x--- 2 jira1 jira1 4096 ápr 16 21.02 hsperfdata_jira1
drwxr-x--- 2 jira2 jira2 4096 ápr 16 21.02 hsperfdata_jira2
drwxr-xr-x 2 root root 4096 ápr 17 06.20 hsperfdata_root
drwxr-xr-x 2 spadm staff 4096 ápr 16 21.22 hsperfdata_spadm
drwxr-xr-x 2 spadm staff 4096 ápr 16 09.59 jna-109637933
-rwxr-xr-x 1 confluence confluence 2855772 ápr 17 09.30 kerberods
-rwxr-xr-x 1 confluence confluence 852368 ápr 17 06.50 khugepageds
-rw-r----- 1 confluence confluence 0 ápr 16 03.51 lok
drwxr-x--- 3 confluence confluence 4096 ápr 12 12.16 systemd-private-afjdhdicjijo473skiosoohxiskl573q-systemd-timesyncc.service-g1g5qf
file khugepageds 
khugepageds: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), statically linked, stripped

When i start the instance the start procedure require much cpu and take some minutes

from catalina.out

INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 158709 ms

Could You help me?

thanks

lajos

[arcadiaengineering]

We had the same this morning. This is what I did:

1. Navigate to the directory that has your cron jobs. In my case it was /var/spool/cron 

2. use top to find the PID of khugepageds and kerberods

3. Type:

> confluence; kill KHUGEPAGEDSPID; kill KERBERODSPID

(where KHUGEPAGESPID is the pid of the khugepageds process and KERBERODSPID is the pid of kerberospid)

4. Upgrade Confluence as soon as possible otherwise you'll be hit again in a couple of hours.

[Feher Lajos]

I've made read-only the crontab file and 've been upgraded the confluence to the latest release (5.15).

Unfortunately the start procedure still slow 

"INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 169533 ms"

CPU consuming

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                              

31415 confluen  20   0 3279m 129m  19m S 125.0  0.9   0:03.77 java                                                                                                                 

29221 confluen  20   0 3944m 1.6g  43m S 28.8 11.0   5:53.73 java                                                                                                                  

And when i want to edit the page, the editor not loading and i immediately got this message:

"This page is taking longer to load then usual.
Give it a few moments, then try refreshing.
Sill having issues? Contact you Confluence admin."

Any further idea?

thanks

lajos

[Feher Lajos]

Sorry, i misspelled the correct Confluence version 6.15.2

[Feher Lajos]

i resolved my problem.

I've deleted all files and directories from /tmp.

After delete, the Confluence instance start and i'm able to edit pages again.

Thanks for helps!

lajos

[JP _AC Bielefeld Leader_]

Did you check out the latest security advisories from Atlassian? What version are you on?

https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html