Confluence isn't starting

[Sean Terry]

We've inherited an old Jira+Confluence installation, and while we managed to migrate to Jira in the cloud several months ago, migrating Confluence has been a prolonged affair.

And now we get the following when trying to access confluence:

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.
Apache/2.4.7 (Ubuntu) Server at 172.31.16.123 Port 80

The contents of atlassian-confluence.log are fairly sparse:

2019-05-03 15:10:38,516 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 5.5 [build 5501 based on commit hash a39771cd4bb7ab00a226a7ea8d6dd65e64226a7f]
2019-05-03 15:10:41,680 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2019-05-03 15:10:45,831 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager

 Also including the catalina log file:

May 03, 2019 3:10:22 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
May 03, 2019 3:10:22 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'minProcessors' to '5' did not find a matching property.
May 03, 2019 3:10:22 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'maxProcessors' to '75' did not find a matching property.
May 03, 2019 3:10:22 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
May 03, 2019 3:10:22 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'useURIValidationHack' to 'false' did not find a matching property.
May 03, 2019 3:10:22 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.
May 03, 2019 3:10:22 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
May 03, 2019 3:10:22 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
May 03, 2019 3:10:23 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8200"]
May 03, 2019 3:10:23 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1055 ms
May 03, 2019 3:10:23 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Tomcat-Standalone
May 03, 2019 3:10:23 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.47

Apache log is full of this kind of information (client IP addresses removed):

[Fri May 03 15:15:01.121351 2019] [proxy:error] [pid 1182:tid 140526053467904] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8100 (localhost) failed
[Fri May 03 15:15:01.121351 2019] [proxy:error] [pid 1184:tid 140526087038720] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8100 (localhost) failed
[Fri May 03 15:15:01.121413 2019] [proxy:error] [pid 1182:tid 140526053467904] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 60s
[Fri May 03 15:15:01.121425 2019] [proxy:error] [pid 1184:tid 140526087038720] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 60s
[Fri May 03 15:15:01.121441 2019] [proxy_http:error] [pid 1182:tid 140526053467904] [client x:38689] AH01114: HTTP: failed to make connection to backend: localhost
[Fri May 03 15:15:01.121444 2019] [proxy_http:error] [pid 1184:tid 140526087038720] [client x:15747] AH01114: HTTP: failed to make connection to backend: localhost
[Fri May 03 15:15:01.121526 2019] [authz_core:error] [pid 1184:tid 140526087038720] [client x:15747] AH01630: client denied by server configuration: /data/htdocs/static
[Fri May 03 15:15:01.121526 2019] [authz_core:error] [pid 1182:tid 140526053467904] [client x:38689] AH01630: client denied by server configuration: /data/htdocs/static
[Fri May 03 15:15:01.250698 2019] [proxy:error] [pid 1184:tid 140526078646016] AH00940: HTTP: disabled connection for (localhost)
[Fri May 03 15:15:01.250762 2019] [authz_core:error] [pid 1184:tid 140526078646016] [client x:29414] AH01630: client denied by server configuration: /data/htdocs/static
[Fri May 03 15:15:01.403357 2019] [proxy:error] [pid 1184:tid 140526070253312] AH00940: HTTP: disabled connection for (localhost)
[Fri May 03 15:15:01.403424 2019] [authz_core:error] [pid 1184:tid 140526070253312] [client x:39328] AH01630: client denied by server configuration: /data/htdocs/static
[Fri May 03 15:15:01.404956 2019] [proxy:error] [pid 1184:tid 140526053467904] AH00940: HTTP: disabled connection for (localhost)
[Fri May 03 15:15:01.405017 2019] [authz_core:error] [pid 1184:tid 140526053467904] [client x:19249] AH01630: client denied by server configuration: /data/htdocs/static
[Fri May 03 15:15:01.411412 2019] [proxy:error] [pid 1184:tid 140526045075200] AH00940: HTTP: disabled connection for (localhost)
[Fri May 03 15:15:01.411480 2019] [authz_core:error] [pid 1184:tid 140526045075200] [client x:43137] AH01630: client denied by server configuration: /data/htdocs/static

 I think port 8100 is old jira and 8200 is confluence. Seeing things going to 8100 when I'm trying to access confluence makes me suspicious, but being a total Linux noob, I don't know how to proceed. There have been no system or configuration changes made that I am aware of. Reboots have had no effect.

Does anyone have insight on what I can try?

[Daniel Eads · Accepted solution]

Hey Sean,

Actually based on a few lines in your log, I think it's possible your Confluence instance might be infected with malware due to a security advisory from a couple months ago. I've written a detailed article on how to work on detection for this and steps you can take for remediation.

The things that tipped me off that this is likely the case from your logs:

• INFO: Starting Servlet Engine: Apache Tomcat/7.0.47

  This tells me you're running on a vulnerable version of Confluence (based on the version of Tomcat server)

• Setting property 'debug' to '0' did not find a matching property.

  While it's not a definitive "you've been affected", we do see this a lot in instances that were attacked due to the CVE

Let me know if you have questions after reading the article!

Thanks,
Daniel | Atlassian Support

[Sean Terry]

Thanks, Daniel. Following those instructions, I was able to get confluence back online for the time being.

[Moses Thomas]

@Sean Terry   Have you managed to  migrate your Jira /Confluence server  to  cloud ? 

if yes  then should be accessing Jira/confluence  with  a different  Url:   for instance  https://xxx.atlassian.net/wiki    but if not  then  you should try  to  check   the confluence  /jira  process   in ubuntu. If  there is   a  process,  then  kill  it  and  start jira/confluence  again.

then access jira/confluence  again.

Hope this helps.