Confluence works for a few minutes and then stops

[Herman Ronk]

After years of functioning properly our instance became unstable this week. We run Jira and Confluence on one server and use a Apache proxy.

Jira is still working fine, but Confluence stops after about 3 minutes of running. From then on the port (8443) stops listening and the proxy returns a 503 error. The Confluence service also keeps running

There is no apparent error in the cataline.out logging and besides the obvous error (503) the apache logging also gives no clue about what is happening.

Server.xml

<Server port="8000" shutdown="SHUTDOWN" debug="0">
<Service name="Tomcat-Standalone">

<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
maxThreads="148" minSpareThreads="10"
enableLookups="false" acceptCount="20" debug="0" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"
proxyName="***.nl" proxyPort="443" scheme="https"/>


<Engine name="Standalone" defaultHost="localhost" debug="0">

<Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">
<Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">
<Manager pathname="" />
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context>

<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0" reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context>
</Host>

</Engine>

</Service>
</Server>

Apache vhost

<IfModule mod_ssl.c>
<VirtualHost _default_:443>
ServerName ***.nl
# JIRA Proxy Configuration:
<Proxy *>
Order deny,allow
Allow from all
</Proxy>

SSLProxyEngine On
ProxyRequests Off
ProxyPreserveHost Off
ProxyPass / http://localhost:8090/ retry=0 connectiontimeout=600 timeout=600 nocanon
ProxyPassReverse / http://localhost:8090/
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/***.nl/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/***.nl/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/***.nl/fullchain.pem

SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"

</VirtualHost>
</IfModule>

catalina.out

18-Apr-2019 21:19:55.700 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:55.805 INFO [http-nio-8090-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:55.907 INFO [http-nio-8090-exec-7] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:56.589 INFO [http-nio-8090-exec-3] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:56.896 INFO [http-nio-8090-exec-10] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:56.964 INFO [http-nio-8090-exec-10] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
18-Apr-2019 21:19:56.966 INFO [http-nio-8090-exec-10] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorGrammarsSupport
18-Apr-2019 21:19:56.966 INFO [http-nio-8090-exec-10] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport
18-Apr-2019 21:19:57.227 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:57.252 INFO [http-nio-8090-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:58.511 INFO [http-nio-8090-exec-2] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:58.914 INFO [http-nio-8090-exec-11] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'

apache error

[Thu Apr 18 21:20:04.086100 2019] [proxy:error] [pid 778:tid 139752279807744] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8090 (localhost) failed
[Thu Apr 18 21:20:04.086153 2019] [proxy:error] [pid 778:tid 139752279807744] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
[Thu Apr 18 21:20:04.086165 2019] [proxy_http:error] [pid 778:tid 139752279807744] [client ****:15087] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://***

apache

orga.duh-events.nl:443 *** - - [18/Apr/2019:21:20:02 +0200] "POST /synchrony-proxy/v1/data/Synchrony-afecb7c1-86dd-361c-bf94-86f370acf8a8/confluence-3342744?state-at=@head&state-format=type-tagged&rewrite-request=true&cached=true HTTP/1.1" 200 31 "https://***" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.48 Safari/537.36 Edg/74.1.96.24"
orga.duh-events.nl:443 *** - - [18/Apr/2019:21:20:02 +0200] "GET ***?src=contextnavpagetreemode HTTP/1.1" 200 31 "https://***" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.48 Safari/537.36 Edg/74.1.96.24"
orga.duh-events.nl:443 *** - - [18/Apr/2019:21:20:03 +0200] "POST /synchrony-proxy/v1/data/Synchrony-afecb7c1-86dd-361c-bf94-86f370acf8a8/confluence-3342744?state-at=@head&state-format=type-tagged&rewrite-request=true&cached=true HTTP/1.1" 503 632 "https://***" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.48 Safari/537.36 Edg/74.1.96.24"
orga.duh-events.nl:443 *** - - [18/Apr/2019:21:20:04 +0200] "GET ***?src=contextnavpagetreemode HTTP/1.1" 503 784 "https://***" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.48 Safari/537.36 Edg/74.1.96.24"

Normaly I would suspect the database, but it is running just fine (as is Jira). 

Does anybody have a clue about what the problem might be? Or where to look for more logging to find the source of the problem?

[Bill Bailey]

There is an earlier security alert that hit me yesterday, and did the same thing, 503.

https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html

After you start confluence, you can run the following to see what other processes are being kicked off (assuming confluence is the user name running confluence).

top -u confluence

I had to manually edit the crontab associated with confluence to get rid of the issue.

[Administrator AchtS]

Any solution, i've gut the same Problem

[Michael Boberek]

Hello
I have exactly the same problem.
Also already update to 6.15.2
BR Michael

[Bastian Stehmann]

Hi @Herman Ronk ,

There is currently a vulnerability in confluence that is a lot exploited, maybe you have been hit by that.

[Bastian Stehmann]

https://confluence.atlassian.com/doc/confluence-security-advisory-2019-04-10-968660855.html

[Herman Ronk]

Thanks for the relpy, but this does not seem to be the problem (applied the fix anyway, but no change in the situation)