Confluence crashes constantly

[Christoph Becker]

Out Confluence installation (version 6.4.0, self-hosted on Linux-Debian 8) keeps crashing virtually at random.

The catalina.log prints:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f7ca31fef52, pid=10943, tid=0x00007f7ca0fd1700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x5c8f52]  G1UpdateRSOrPushRefOopClosure::do_oop(unsigned int*)+0x42
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid10943.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

I can also paste the contents of the "/tmp/hs_err_pid10943.log" file, but it's rather large.

This keep happening without any warnings, while people use Confluence. Sometimes it even crashes withing 5 minutes of restarting Confluence.

[Mike Rathwell]

Hi, @Christoph Becker

I think the suggestion that @Igor M_ is headed the right direction. This feels like an OoM issue.

While the system you are running this on has 7GB RAM, that does NOT mean that you've allocated it to Java. The default values in the setenv.sh script are pretty small. Given that you have both on the system, I'd suggest you set both Xms and Xmx at 2048m (same vernacular as used in the existing setenv.sh). Setting both the same was a tip I found SOMEwhere here (or on Atlassian pages) to help reduce the impact of garbage collection.

As to the suggestion of upgrading Java that @Danyal Iqbal put out... that is never a Bad Idea. Unless the Atlassian supported platform pages have a specific Java version cited that gives problems, you're probably good to do that. I do that myself. That said, this doesn't feel like a Java bug.

So... go in and edit your Java memory values RIGHT NOW. (I am assuming a Linux host but if you are on Windows, let me know and I can help you with that as well). Then, either bounce the Confluence instance when next is a good time OR it'll automagically get picked up the next time it falls over and you have to restart.

[Christoph Becker]

I changed the Xms and Xmx as you suggested, however now confluence wont start at all:

22-Jun-2018 19:11:58.858 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file
22-Jun-2018 19:11:58.859 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
22-Jun-2018 19:12:23.295 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
22-Jun-2018 19:12:23.295 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.github.gquintana.metrics.sql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
22-Jun-2018 19:12:23.295 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.hsqldb.jdbc.JDBCDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
22-Jun-2018 19:12:23.296 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [net.sourceforge.jtds.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
22-Jun-2018 19:12:23.296 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
22-Jun-2018 19:12:23.296 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
22-Jun-2018 19:12:23.300 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [AtlassianEvent::CustomizableThreadFactory-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:748)

[Mike Rathwell]

Christoph,

It SEEMS like you might have ended up with a syntax error in the setenv.sh script....

Make sure it is exactly -Xms2048m and -Xmx2048m (no spaces or anything like that). I've done that before when I borked those and it basically just didn't know where to go with it.

I just checked my environment to make sure I wasn't talking out my ... hat... on that one. Failing that and if there just isn't enough memory free on your system as it stands, scale 'em back to 1024m (both values which are both larger than the defaults)

[Christoph Becker]

I was able to change the Xmx and Xms values last week (from 1024 to 2048), however this didn't seem to have had any effect.

I already experienced four more random crashes this week

[Danyal Iqbal]

backup your system, upgrade java or confluence :) and retry.

Debugging this issue would be a royal pain in the ****.

[Christoph Becker]

I'm using the bundled JRE. Isn't it more likely to cause issues, if I install another java version?

[AnnWorley]

Danyal's suggestion is conventional wisdom for sigsev faults as they do indicate crashes in the underlying Java. Please see: Segfault error - Confluence unavailable due to Java crash

[Danyal Iqbal]

I admire your persistence @Christoph Becker!

If you really wanna debug this, have a look  at the dump.--e.g

/tmp/hs_err_pid10943.log

 You should be able to see the library causing this behaviour in this log . However, there is not much you can do about it. You will  have to upgrade/downgrade java in any case:) but you will be smarter never the less.

Happy Debugging:)

[Igor M.]

Hello,

SIGSEGV could have any number of causes.

I suggest starting with check for any OutOfMemeory errors in catalina.out and atlassian-confluence.log, if you can see those around the time you encounter crash, try following How to fix out of memory errors by increasing available memory guide and increasing or decreasing your xms and xmx values, as running out of memory could either mean, Confluence does not have enough memory to run within allocated space or Confluence service does not have enough memory to run on your server.

Let me know how it goes

Regards,

Igor

[Christoph Becker]

Thanks for the reply.

I searched through the atlassian-confluence.log, however I can't find any OutOfMemeory errors. The System has 7GB RAM, however it runs both Confluence AND Jira. However Jira always runs fine without any crashes.

The atlassian-confluence.log is a bit long, thats why I pasted it here:  https://pastebin.com/HYxRgP40