Confluence stopps working after setup and first restart due to memory issues

[Schmeil]

Hi!

I did install confluence server using atlassian-confluence-7.10.2-x64.bin on a Linux Debian Buster with postgresql 10 data base.

I performed the setup without any issues including connecting to the data base, creating an admin user and set up a test space, then I was able to use confluence. But after stopping and starting confluence via /etc/init.d/confluence1 start it doesnt come up again. After some minutes high cpu usage and using >2gb RAM confluence is stopping by exception.

The logs show "java.lang.OutOfMemoryError", some experiments with variables (eg. -Xms300m -Xmx300m and -XX:ReservedCodeCacheSize=2048m) did not help.

Why the hell confluence stops working after instaling???

The machine has 16GB RAM and 300GB SSD storage, with debian linux 10

[Thiago Masutti · Accepted solution]

Hi @Schmeil 

Although the JVM is throwing an Out of Memory error, it seems it is not complaining about the heap space, so changing Xmx and Xms values may not be helpful in this case.

Sometimes the JVM can throw this error when it is unable to create new threads if the OS user has reached the maximum number of processes.

Would you mind login in the OS terminal as the user running the Confluence processes and sharing the output of the following command?

ulimit -a

If you are running Confluence as a service in Linux, it might be the case where the service is constrained to a certain number of processes/tasks.

Take a look at this SuSE KB article on changing the maximum number of tasks/threads/processes allowed by a service.

Let us know if this helps.

Kind regards,
Thiago

[Schmeil]

for the confluence user the output is the following:

$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 0
memory(kbytes) unlimited
locked memory(kbytes) 65536
process 62987
nofiles 1024
vmemory(kbytes) unlimited
locks unlimited
rtprio 0

[Thiago Masutti]

Can you send the Confluence service definition as well?

[Schmeil]

Hi @Thiago Masutti , what do you mean by this exactly ?

[Thiago Masutti]

If you are using init.d, there will be a file /etc/init.d/<service name>.

If you are using systemd, there will a .conf file within the /etc/systemd/system/<service name>.service.d/ folder.

Either of these two will be your service definition.

[Schmeil]

#!/bin/bash

### BEGIN INIT INFO
# Provides: confluence
# Required-Start: $remote_fs $syslog $time $named
# Required-Stop: $remote_fs $syslog $time $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Atlassian Confluence
# Description: Atlassian Confluence Server
### END INIT INFO

# Confluence Linux service controller script
cd "/srv/confluence/bin"

case "$1" in
start)
./start-confluence.sh
;;
stop)
./stop-confluence.sh
;;
restart)
./stop-confluence.sh
./start-confluence.sh
;;
*)
echo "Usage: $0 {start|stop|restart}"
exit 1
;;
esac

[Thiago Masutti]

Thanks.

So your service definition is standard.

The number of processes allowed to the user seems fine.

Now I'm concerned with the number of open files.

As root, edit the /etc/security/limits.conf file and add the following entries:

<os_user_running_confluence> soft nofile 32768
<os_user_running_confluence> hard nofile 32768

Try to start the Confluence service and let us know if the same error occurs.

Regards,
Thiago

[Schmeil]

SOLVED!   Thank you!

For resolution I performed those steps - maybe one of the solved the problem already:

• confluence soft nofile 32000
  confluence hard nofile 64000
• correction of the postgresDb (dropped and recreated)
  • was: Collate C and ctype C (due to wrong suggestion from JIRA installer doc)
  • new: all UTF8
• vim /etc/systemd/system.conf
  [Manager]
  DefaultTasksMax=15288
• reboot

btw. I did reset the OPTIONS to default values: 

 /srv/confluence/jre//bin/java -Djava.util.logging.config.file=/srv/confluence/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 -Dconfluence.context.path= -Datlassian.plugins.startup.options= -Djava.locale.providers=JRE,SPI,CLDR -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx1024m -XX:+UseG1GC -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/srv/confluence/logs/gc-2021-01-22_00-49-37.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/srv/confluence/logs/gc-2021-01-22_00-49-37.log::filecount=5,filesize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -XX:+IgnoreUnrecognizedVMOptions -Dignore.endorsed.dirs= -classpath /srv/confluence/bin/bootstrap.jar:/srv/confluence/bin/tomcat-juli.jar -Dcatalina.base=/srv/confluence -Dcatalina.home=/srv/confluence -Djava.io.tmpdir=/srv/confluence/temp org.apache.catalina.startup.Bootstrap start

Consulted docs:

• https://docs.oracle.com/en/java/javase/11/tools/java.html#GUID-3B1CE181-CD30-4178-9602-230B800D4FAE
• https://stackoverflow.com/questions/5090858/how-do-you-change-the-character-encoding-of-a-postgres-database
• https://stackoverflow.com/questions/12079087/fork-retry-resource-temporarily-unavailable
• https://www.thegeekdiary.com/understanding-etc-security-limits-conf-file-to-set-ulimit/

[Nic Brough -Adaptavist-]

Where are you setting the memory variables?  Actually, only the "max heap memory" one matters much here - the others you've mentioned have performance impacts and could throw other errors, but they won't lead to "out of memory".

-Xmx300m is the right format (I mention this because some people seem to think it is case insensitive - it's not), and although 300m is too low for Confluence, your next test with 8000m should have worked, which makes me suspect that your numbers might simply not be being applied.

When you get the error, could you look back up through the log to the point where it started?  Confluence echoes out the memory settings it has ended up running with and my guess is that it's not inherited the one you have set for some reason.

[Schmeil]

Hi, as far I know Xms is minimum and Xmx maximum heap size. Thats why I tested it with this set parameters - unfortunately without a solution :

Using CATALINA_OPTS: -XX:ReservedCodeCacheSize=2000m -XX:+UseCodeCacheFlushing -Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 -Dconfluence.context.path= -Datlassian.plugins.startup.options='' -Djava.locale.providers=JRE,SPI,CLDR -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx8000m -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/srv/confluence/logs/gc-2021-01-22_11-44-25.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/srv/confluence/logs/gc-2021-01-22_11-44-25.log::filecount=5,filesize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -XX:+IgnoreUnrecognizedVMOptions

Could it be that there are other java / memory parameters needs to be changed on the machine?

[Nic Brough -Adaptavist-]

What does the application log say the parameters are coming out as when you start the applicaiton?

[brbojorque]

Hi @Schmeil ,

300mb memory does not cut it in Confluence. Try these variables

-Xms8000m -Xmx8000m

And update this variable XX:ReservedCodeCacheSize to the default value whatever it is.

[Schmeil]

thank you.. unfortunately those parameters doesnt help.. I still see such log messages after starting:

[93,640s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.Exception in thread "Navlink Plugin Executor:thread-2" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.base/java.lang.Thread.start0(Native Method) at java.base/java.lang.Thread.start(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)

[brbojorque]

Hi @Schmeil ,
May I know the full path of the file you've updated the JVM values with?

Check also the available memory by typing `free -h`

[Schmeil]

Hi, the file is: /srv/confluence/bin/setenv.sh.  

Changes in this file are effectively, e.g.  trying to set the "JVM_MAXIMUM_MEMORY" variable (which can be set in JIRA) leads to error messages in the log.

Starting confluence with used 700MB RAM (of 16GB max.) on the machine ends with used 3,5 GB used after 1,5 minutes .. at this time lot of error messages like this start appearing:

[95,394s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.

Also this message is displayd:

Exception in thread "Catalina-utility-2" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

free -h says:

total used free shared buff/cache available
Mem: 16Gi 3,5Gi 10Gi 69Mi 2,2Gi 12Gi