Sometimes confluence service is down.

[Guillermo Saenz de Santa Maria Hernández]

Sometimes during the night the confluence service goes down. we must to startup again. there is no error indicating the problem.

[Nic Brough -Adaptavist- · Accepted solution]

Are there really no errors in the application or server logs?  When it dies, what are the last few records in <confluence home>/log/atlassian-confluence.log and <confluence install>/logs/catalina.out ?

[Guillermo Saenz de Santa Maria Hernández]

Thanks for your time and excuse me for my first and short question...I hope give you more info:

OS: CentOS 7.7

Confluence version: 6.14.1

atlasasian-conflucence.log is now overwritten.. but we saved catalina.out in last previous restart at 8th of May...  the first error message could be these... I cut the messages and put the first three errors:

08-May-2020 03:50:46.909 INFO [localhost-startStop-4] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.shutdown Shutting down ExecutorService 'defaultSockJsTaskScheduler'
2020-05-08 03:51:06
Full thread dump OpenJDK 64-Bit Server VM (25.192-b12 mixed mode):

"Gemini Blueprint context shutdown thread 1" #543 daemon prio=5 os_prio=0 tid=0x00007f21a0001000 nid=0x79b7 in Object.wait() [0x00007f218bffb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1465)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
- locked <0x00000000c78cd958> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at net.sf.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:36)
at com.atlassian.h

"FelixShutdown" #542 daemon prio=5 os_prio=0 tid=0x00007f219834f800 nid=0x79b6 in Object.wait() [0x00007f21e9e23000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.eclipse.gemini.blueprint.extender.internal.util.concurrent.Counter.waitFor(Counter.java:160)
- locked <0x00000000e9602778> (a org.eclipse.gemini.blueprint.extender.internal.util.concurrent.Counter)
at org.eclipse.gemini.blueprint.extender.internal.util.concurrent.Counter.waitForZero(Counter.java:136)
- locked <0x00000000e9602778> (a org.eclipse.gemini.blueprint.extender.internal.util.concurrent.Counter)
at org.eclipse.gemini.blueprint.extender.internal.util.concurrent.RunnableTimedExecution.execute(RunnableTimedExecution.java:100)
at org.eclipse.gemini.bluep

"localhost-startStop-3" #540 daemon prio=5 os_prio=0 tid=0x00007f2228490800 nid=0x79af in Object.wait() [0x00007f21789d2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.felix.framework.util.ThreadGate.await(ThreadGate.java:79)
- locked <0x00000000c709aea0> (a org.apache.felix.framework.util.ThreadGate)
at org.apache.felix.framework.Felix.waitForStop(Felix.java:1075)
at com.atlassian.plugin.osgi.container.felix.FelixOsgiContainerManager.stop(FelixOsgiContainerManager.java:442)
at com.atlassian.plugin.osgi.container.felix.FelixOsgiContainerManager.onShutdown(FelixOsgiContainerManager.java:247)
at sun.reflect.NativeMethodAcc

Some java errors and finnally this warnings:

08-May-2020 04:06:26.512 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
08-May-2020 04:06:26.618 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
08-May-2020 04:06:26.789 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.
08-May-2020 04:06:26.794 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
08-May-2020 04:06:26.833 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching prope
rty.
08-May-2020 04:06:26.857 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching prope
rty.

Finally the log ends with some info messages:

08-May-2020 04:06:30.818 INFO [localhost-startStop-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servl
et.resource.DefaultServletHttpRequestHandler]
08-May-2020 04:06:30.872 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationConte
xt for namespace 'dispatcher-servlet': startup date [Fri May 08 04:06:29 CEST 2020]; root of context hierarchy
2020-05-08 04:06:30,965 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.14.1 [build 7901 based on commit hash 70b99a3474165a6093a2423f21069b9c63f07abc] -
synchrony version 2.1.0-master-9d112c9d
08-May-2020 04:06:30.984 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy
.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()
08-May-2020 04:06:30.986 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.
http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)
08-May-2020 04:06:31.108 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
08-May-2020 04:06:31.142 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1596 ms

The next time we will save the atlassian-confluence log,... is there any way to upload the zip file generated?

Thanks for your time

[Guillermo Saenz de Santa Maria Hernández]

I have get this error from atlassian-confluence.log before restart the app.

2020-05-22 03:23:00,246 ERROR [Caesium-1-4] [migration.agent.queue.QueueBroker] error An error occurred when getting the next batch for consumer type: USERS_MIGRA
TION. Message: javax.persistence.PersistenceException: Failed to update database schema
com.atlassian.util.concurrent.LazyReference$InitializationException: javax.persistence.PersistenceException: Failed to update database schema

[Guillermo Saenz de Santa Maria Hernández]

I have followed this note and the problem seems to be resolved.

https://confluence.atlassian.com/confkb/confluence-upgrade-failed-with-liquibase-exception-lockexception-could-not-acquire-change-log-lock-error-969527207.html

thanks for your help.

[Guillermo Saenz de Santa Maria Hernández · Accepted solution]

I have followed this note and the problem seems to be resolved.

https://confluence.atlassian.com/confkb/confluence-upgrade-failed-with-liquibase-exception-lockexception-could-not-acquire-change-log-lock-error-969527207.html

thanks for your help.

[repi · Accepted solution]

@Guillermo Saenz de Santa Maria Hernández 

On which OS basis does the Confluence server run?
Some Windows machines have an automatic update activated.
Since Windows systems usually reboot after an update and the service for Confluence either starts too early or is set to manual start, the Confluence does not restart.

[Guillermo Saenz de Santa Maria Hernández]

I have got this error from atlassian-confluence.log before restart the app.

2020-05-22 03:23:00,246 ERROR [Caesium-1-4] [migration.agent.queue.QueueBroker] error An error occurred when getting the next batch for consumer type: USERS_MIGRA
TION. Message: javax.persistence.PersistenceException: Failed to update database schema
com.atlassian.util.concurrent.LazyReference$InitializationException: javax.persistence.PersistenceException: Failed to update database schema

[Guillermo Saenz de Santa Maria Hernández]

Please, could you suggest some tips for this?