Confluence not working at all

[Ezequiel]

Hey guys,

All of a sudden, Confluence (latest server version) stopped working and we can't bring it back to life.

Any ideas? The error we see in the logs is:

19-Sep-2017 20:50:36.211 SEVERE [http-nio-8090-exec-1] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
 org.apache.jasper.JasperException: java.lang.IllegalStateException: getOutputStream() has already been called for this response
 at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:560)
 at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:471)
 at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
 at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
 at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
 at com.atlassian.confluence.util.profiling.ConfluenceActivityFilter.doFilter(ConfluenceActivityFilter.java:39)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
 at com.atlassian.confluence.web.filter.HttpRequestStatsFilter.doFilter(HttpRequestStatsFilter.java:47)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
 at com.atlassian.gzipfilter.GzipFilter.doFilter(GzipFilter.java:96)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
 at com.atlassian.confluence.web.filter.ConfluenceTimingFilter.doFilter(ConfluenceTimingFilter.java:46)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
 at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:716)
 at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:584)
 at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:523)
 at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:436)
 at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:301)
 at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:396)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:176)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
 at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)
 at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1533)
 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1489)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: getOutputStream() has already been called for this response
 at org.apache.catalina.connector.Response.getWriter(Response.java:564)
 at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:212)
 at javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:109)
 at javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:109)
 at com.atlassian.confluence.web.filter.DebugFilter$LoggingResponseWrapper.getWriter(DebugFilter.java:111)
 at javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:109)
 at com.atlassian.confluence.web.filter.DebugFilter$LoggingResponseWrapper.getWriter(DebugFilter.java:111)
 at org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:115)
 at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:108)
 at org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:176)
 at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:120)
 at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:75)
 at org.apache.jsp._500page_jsp._jspService(_500page_jsp.java:944)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
 at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
 ... 67 more

Thanks,

Ezequiel

[Leandro · Accepted solution]

Hi Ann, I'm Ezequiel's colleague. I'd like to thank you for the links provided, however we were unable to fix confluence following those.

We managed to get confluence working, though, here's what I did:

- Create a mysql dump of confluence DB's

mysqldump -h <hostname> -u <user> -p <password> --database <dbname> > <whatever-name-you-like>.sql

 - Drop the old confluence DB. Create a new DB and grant privileges to the user, as if it was a new installation, cause that's what we're gonna do.

mysql -h <hostname> -u <user> -p <password>
drop database <dbname>;
create database confluence;
create user 'confluence'@'%' identified by '<password>';
grant all privileges on confluence.* to confluence;
exit

- Launch a new instance and install confluence on it. We installed 6.4.1. Don't terminate the old instance, you will need stuff from it!!

Follow this link to install confluence

- Stop confluence and connect to the DB. Drop the new database Confluence used for the installation and source the dump from step #1.

mysql -h <hostname> -u <user> -p <password>
drop database confluence;
create database confluence;
source confluence-dump.sql

[Leandro]

- After that, we got the following error:

Confluence will not start up because the build number in the home directory [7401] doesn't match the build number in the database [7400].

This link will help you fix build number.

Solution #1 worked for us. 

- Then we had collation and character set issues. Several of them. 

Here's how to fix collation and character set issues

- Finally, we wanted all the images, thumbnails, attachments and etc we had uploaded. 

Go to:

/var/atlassian/application-data/confluence

And make a backup of attachments/ and thumbnails/ directory from the old instance.

Move all this content to the new confluence instance and just uncompress it on the same path.

- Start confluence. It worked like a charm.

[Ezequiel]

Awesome!!

[Shashidar]

Hi Ann,

May i know the reason for this error in logs?

Even i'm seeing the same error in logs but my production instance is running. How can we resolve the error?

Regards,

[AnnWorley]

Hi Shashidar,

Which error are you seeing in which log? We discussed several errors on this thread. One was for displaying the error page, one was not being able to ping the synchrony proxy and one was concerned with database character set.

It may be better if you post a separate question so we can keep your issue distinct from this case.

Thanks,

Ann

[Ezequiel]

Hi Ann,

I was unable to paste the log, so here's a link to it:

https://www.dropbox.com/s/t0k7gyvmdqf9ywv/atlassian-confluence.log?dl=0

 
The weirdest thing I see is:

2017-09-21 14:38:49,623 WARN [ListenableFutureAdapter-thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] pollHealthcheck Could not ping the synchrony-proxy [http://127.0.0.1:8090/synchrony-proxy/healthcheck]: {}
java.net.SocketTimeoutException: Read timed out

[AnnWorley]

Thanks very much for the log, it includes the whole startup, as I hoped.

I was surprised to see this at the end of the log:

2017-09-21 14:39:12,007 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve

That usually indicates that Confluence is available in the browser, although what is displayed there can turn out to be an error message.

Synchrony Proxy

Like you, I noticed the long stack trace related to the Synchrony health check. My expectation from that error would be issues with collaborative editing. For a useful diagram of how Synchrony works from a network perspective, please see: Administering Collaborative Editing under Proxies.

Non-recommended character set

I was concerned to see this in the log right after start-up, as I have seen Confluence fail to start as a result of character set issues:

2017-09-21 14:37:42,635 WARN [localhost-startStop-1] [confluence.setup.dbcheck.MySQLChecker] checkCharacterSetAndCollation Your database uses non-recommended character set: latin1. See https://confluence.atlassian.com/x/IrYC

Please follow this guide to make sure the right character set is used in the Confluence database: How to Fix the Collation and Character Set of a MySQL Database

[Ezequiel]

Thanks Ann! Will check and let you know.

[AnnWorley]

Hi Ezequiel,

Please let me know more about when the instance stopped working. Did it slow down so that you restarted it and it didn't come back up? Or was the service stopped before it was restarted?

It looks like the log entry you posted is from the logs subdirectory of the Confluence installation directory. (It is often called catalina.out.) That log is for the underlying web server and will let us know if there are Tomcat configuration or other basic issues. The error you posted seems to indicate that Tomcat cannot display an error page, which is probably not the root cause of the outage.

In the Confluence home directory there is another logs subdirectory, which contains the atlassian-confluence.log. In that log, please search for Starting Confluence and then scroll down to any errors that Confluence reports during the startup. 

If the errors aren't informative enough, please post them on this thread. If I am not familiar with them I will be happy to research them and let you know my findings.

Thanks,

Ann