Confluence service went down in the server, and not able to come up.

[Rahul Singhal]

Hi,

I have checked in my server that confluence service went down on 10 Nov and when I try to start it it is not coming up neither it is recording anything in logs. The last logs was on 10 Nov after that nothing is recorded.

Logs-

2018-11-10 23:12:35,266 WARN [Resource Destroyer in BasicResourcePool.close()] [mchange.v2.resourcepool.BasicResourcePool] log Failed to destroy resource: com.mchange.v2.c3p0.impl.NewPooledConnection@33ee8a00
java.lang.NegativeArraySizeException
at oracle.net.ano.CryptoNIONSDataChannel.readDataFromSocketChannel(Unknown Source)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:62)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:908)
at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:2005)
at com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:642)
at com.mchange.v2.c3p0.impl.NewPooledConnection.closeMaybeCheckedOut(NewPooledConnection.java:255)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:622)
at com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:1076)
at com.mchange.v2.resourcepool.BasicResourcePool.destroyResource(BasicResourcePool.java:1101)
at com.mchange.v2.resourcepool.BasicResourcePool.destroyResource(BasicResourcePool.java:1062)
at com.mchange.v2.resourcepool.BasicResourcePool.access$100(BasicResourcePool.java:44)
at com.mchange.v2.resourcepool.BasicResourcePool$5.run(BasicResourcePool.java:1316)
2018-11-10 23:12:35,290 INFO [localhost-startStop-2] [com.atlassian.confluence.lifecycle] shutdownCacheManager Shutting down EhCache cache manager
2018-11-10 23:12:35,295 INFO [localhost-startStop-2] [com.atlassian.confluence.lifecycle] contextDestroyed Stopping Confluence
2018-11-10 23:12:35,334 INFO [localhost-startStop-2] [confluence.cluster.hazelcast.HazelcastClusterManager] stopCluster Shutting down the cluster

Catalina.out logs

10-Nov-2018 23:30:43.346 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Caesium-1-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.takeUnderLock(SchedulerQueueImpl.java:292)
com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.take(SchedulerQueueImpl.java:246)
com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:53)
com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
java.lang.Thread.run(Thread.java:745)
10-Nov-2018 23:30:43.351 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Caesium-1-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.takeUnderLock(SchedulerQueueImpl.java:292)
com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.take(SchedulerQueueImpl.java:246)

[Yoga Reddy {Appfire}]

Hi @Rahul Singhal,

Can you restart the Confluence and share the latest logs (atlassian-confluence.log)? What is the Confluence version and the database you are using? 

Please verify if you have free memory available on the server along with the sufficient space. Reverify all the permissions to the installation and home directory. 

Thanks

Yoga