Confluence crashes multiple times per day

[Mark Anderson]

We have been running confluence since 2014 and love the tool.  We were running v.5.9.4 and had no problems for a very long time.  Recently Confluence started crashing multiple times per day so we upgraded to the latest version, 6.8.0, but the issue still occurs.  I have looked through the logs but can't find why it is crashing.  I was hoping someone could help.

I have removed all add-ons except Gliffy, which is up to date.

All instance health checks pass.

The support zip file which includes all the logs etc. can be downloaded at:

[redacted]

Any help is appreciated! 

Thanks

Mark

[AnnWorley]

From your logs it looks like you need to increase your database connection pool from 30 to 60:

2018-03-30 20:16:53,373 WARN [localhost-startStop-1] [confluence.impl.health.DefaultHealthCheckRunner] logEvent Your database connection pool allows 30 connections, which is not enough to support the 48 HTTP threads in your Tomcat configuration. Either increase the pool size to at least 60 connections or reduce the maximum number of HTTP threads to 20 or fewer.

This is a common issue, steps to resolve are documented here: Confluence slows and times out during periods of high load due to database connection pool

In older versions of Confluence the default database connection pool was 30 and we saw this issue frequently, later versions use 60 connections by default but when you upgrade as you did, it preserves your old setting.

I look forward to hearing whether Confluence is stable after you increase the database connection pool and restart Confluence.

[AnnWorley]

Actually, @Suren Raj makes a good point - your db connection pool in confluence.cfg.xml is 200:

 <property name="hibernate.c3p0.max_size">200</property>

at the same time, your threads in server.xml are also 200, (maxThreads="200") when ideally they should be lower:

Confluence slows and times out during periods of high load due to database connection pool:

Limit the number of HTTP threads available to be closer to the number of database connections available. If you are going to have a bottleneck, then the HTTP thread pool is better than the DB connection pool, since fewer resources are consumed while waiting for an HTTP connection.
How to do this depends on how your application server is configured.
For a standalone configuration using the default connector, change (or add if it doesn't exist) the maxThreads attribute of the Connector defined in <conf-install>\conf\server.xml. If this parameter isn't specified it will default to 200.

maxThreads="48"

[Mark Anderson]

Ann,

I tried that a few days ago while I was troubleshooting.  Right now it is at 60.  I tried flushing the cache as Suren suggested and will let you know if that helps out.  Anything in the log files after the 6th of April should be applicable to how everything is currently configured.  Any additional suggestions are appreciated!

[Mark Anderson]

When I flushed the cache I updated the maxThreads to 48 per the information in the link provided.

 


[root@devtools confluence]# vi confluence.cfg.xml 

 

<?xml version="1.0" encoding="UTF-8"?>

 

<confluence-configuration>

  <setupStep>complete</setupStep>

  <setupType>custom</setupType>

  <buildNumber>7701</buildNumber>

  <properties>

    <property name="admin.ui.allow.daily.backup.custom.location">false</property>

    <property name="admin.ui.allow.manual.backup.download">false</property>

    <property name="admin.ui.allow.site.support.email">false</property>

    <property name="atlassian.license.message">[redacted]</property>

    <property name="attachments.dir">${confluenceHome}/attachments</property>

    <property name="confluence.setup.server.id">BIYW-6EUF-H09D-8XNT</property>

    <property name="confluence.webapp.context.path"></property>

    <property name="hibernate.c3p0.acquire_increment">1</property>

    <property name="hibernate.c3p0.idle_test_period">100</property>

    <property name="hibernate.c3p0.max_size">60</property>

    <property name="hibernate.c3p0.max_statements">0</property>

    <property name="hibernate.c3p0.min_size">0</property>

    <property name="hibernate.c3p0.timeout">30</property>

    <property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>

    <property name="hibernate.connection.isolation">2</property>

    <property name="hibernate.connection.password">password</property>

    <property name="hibernate.connection.url">jdbc:mysql://localhost/confluence</property>

    <property name="hibernate.connection.username">confluenceuser</property>

    <property name="hibernate.database.lower_non_ascii_supported">true</property>

    <property name="hibernate.dialect">com.atlassian.confluence.impl.hibernate.dialect.MySQLDialect</property>

    <property name="hibernate.setup">true</property>

    <property name="jwt.private.key">[redacted]</property>

    <property name="jwt.public.key">[redacted]</property>

    <property name="lucene.index.dir">${localHome}/index</property>

    <property name="synchrony.btf">true</property>

    <property name="synchrony.encryption.disabled">true</property>

    <property name="synchrony.proxy.enabled">true</property>

    <property name="webwork.multipart.saveDir">${localHome}/temp</property>

  </properties>

</confluence-configuration>

[root@devtools conf]# cat server.xml 

<Server port="8000" shutdown="SHUTDOWN" debug="0">

    <Service name="Tomcat-Standalone">

        <Connector port="8090" connectionTimeout="20000" redirectPort="8443"

                maxThreads="48" minSpareThreads="10"

                enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"

                protocol="org.apache.coyote.http11.Http11NioProtocol"

proxyName="wiki.example.com" proxyPort="443" scheme="https" secure="true" />

 

        <Engine name="Standalone" defaultHost="localhost" debug="0">

 

            <Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">

                <Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">

                    <!-- Logger is deprecated in Tomcat 5.5. Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties -->

                    <Manager pathname="" />

                    <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />

                </Context>

 

                <Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0" reloadable="false" useHttpOnly="true">

                    <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />

                </Context>

            </Host>

 

        </Engine>

 

        <!--

            To run Confluence via HTTPS:

             * Uncomment the Connector below

             * Execute:

                 %JAVA_HOME%\bin\keytool -genkey -alias tomcat -keyalg RSA (Windows)

                 $JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA  (Unix)

               with a password value of "changeit" for both the certificate and the keystore itself.

             * Restart and visit https://localhost:8443/

 

             For more info, see https://confluence.atlassian.com/display/DOC/Running+Confluence+Over+SSL+or+HTTPS

        -->

<!--

        <Connector port="8443" maxHttpHeaderSize="8192"

                   maxThreads="48" minSpareThreads="25"

                   protocol="org.apache.coyote.http11.Http11NioProtocol"

                   enableLookups="false" disableUploadTimeout="true"

                   acceptCount="100" scheme="https" secure="true"

                   clientAuth="false" sslProtocols="TLSv1,TLSv1.1,TLSv1.2" sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2" SSLEnabled="true"

                   URIEncoding="UTF-8" keystorePass="<MY_CERTIFICATE_PASSWORD>"/>

-->

    </Service>

</Server>

[Mark Anderson]

After changing the maxThreads and max_size Confluence ran for a good part of the day yesterday but was not running when I got in this morning.  I restarted the service at 8:03am this morning then created a new support zip file.  Hopefully that has some breadcrumbs that can help figure out why it keeps crashing.

The support zip file can be downloaded at: [redacted]

Thanks for the help so far I am open to trying any suggestions.

[Mark Anderson]

@AnnWorley and @Suren Raj.  Thanks for helping out so far. We are still having issues with Confluence crashing almost every day.  I have implemented the suggestions above and let it run for a few days then created a new support zip file this morning.  It can be downloaded at:

[redacted]

I am open to trying any suggestions!  Thanks in advance for any help.

[David Benson _draw_io_]

What about setting up a clean Confluence 6.8 instance and do an XML backup and restore https://confluence.atlassian.com/doc/site-backup-and-restore-163578.html?

Maybe even don't go all the way to 6.8, try something like 6.3 first, then upgrade that if it's stable.

[Suren Raj]

Hi, Mark.

One particular error that keeps standing out over and over during startup attempt is failure to reach DB

2018-04-02 09:30:03,861 WARN [C3P0PooledConnectionPoolManager[identityToken->2so1aa9u11bsb4l1ouu6p6|3362772e]-HelperThread-#2] [mchange.v2.resourcepool.BasicResourcePool] log Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@4069550c is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
2018-04-02 09:30:18,864 WARN [C3P0PooledConnectionPoolManager[identityToken->2so1aa9u11bsb4l1ouu6p6|3362772e]-HelperThread-#2] [mchange.v2.resourcepool.BasicResourcePool] log com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@64e12f6b -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor52.newInstance(Unknown Source)

So it seems your instance is stuck trying to establish a connection to DB and keeps failing. Your connection pool configuration is in order so I suspect that connection was not properly severed during the shutdown, or cache wasn't flushed.

• If Confluence is not shut-down properly follow Start and Stop Confluence instructions, please do so.
• Clear plugin cache as stated in the below link:
  
  https://confluence.atlassian.com/confkb/how-to-clear-confluence-plugins-cache-297664846.html

Let us know how it goes.

[Mark Anderson]

Suren,

Thanks for your help!

I shutdown the server and flushed the cache as you suggested.  I believe the log from April 2nd was while I was upgrading the server.  I have since fixed the database connection.  I also tried changing the database connection pool as suggested by Ann.  I will let you know if this solves the issue.