Not able to access confluence

[LokanaSM]

Hi Team,

Yesterday we experienced a loss of our VM server that host our Confluence instance. We have restarted the server and confluence service.

We restarted the DB server with no issue but the confluence application it self is not starting properly below is the error we are getting:

You cannot access Confluence at present. Look at the table below to identify the reasons.

Type         Description                                         Exception     Level     Time
startup     Unable to start up Confluence. Fatal error during startup sequence:             fatal            2018-05-31 13:42:44
        confluence.lifecycle.core:mananagedjobs (Start and stop the Managed Scheduled Jobs)
        - org.springframework.transaction.UnexpectedRollbackException:
          Transaction rolled back because it has been marked as rollback-only

Your server id is: XXXX-XXXX-XXXX-XXXX     

==========Conflunce Details======
Confluence Version :6.2.1
Oracl DB: 12

Please do
Thanks
Sanjeev Kumar N

[Moses Thomas · Accepted solution]

@LokanaSMPlease follow documentation here. make sure you  are doing the right thing.

https://confluence.atlassian.com/doc/database-setup-for-oracle-173821.html

best!

[Sanjeev kumar Narayanaswamy]

Hi Thomas,

Oracle setup is fine and after up-gradation from 5.8.18 to 6.2.1 we where able to access the Confluence for almost more than a year.
Unexpectedly confluence system went down and after restart we are not able to access confluence and getting error .
I have attached screen shot of the atlassian-confluence.log and catalina logs.

Please do the needful.

Thanks
Sanjeev Kumar N

[Sanjeev kumar N]

Hi Thomas,

Able to find some major ERROR in attlassian-confluence.log. Looks like DB issue. Please suggest me.

2018-06-04 12:54:37,614 ERROR [Caesium-1-2] [engine.jdbc.spi.SqlExceptionHelper] logExceptions ORA-01555: snapshot too old: rollback segment number  with name "" too small
ORA-22924: snapshot too old

2018-06-04 12:54:37,615 WARN [Caesium-1-2] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister':
Failed due to UNCLASSIFIED_FAILURE
2018-06-04 12:54:37,617 WARN [Caesium-1-2] [confluence.setup.bandana.ConfluenceDaoBandanaPersister] getObjectFromValue Configuration could not be loaded because class could not be found
 (context: null, key: null).
java.lang.NullPointerException: null
java.lang.NullPointerException
    at java.io.StringReader.<init>(StringReader.java:50)
    at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.getObjectFromValue(ConfluenceDaoBandanaPersister.java:100)
    at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.retrieve(ConfluenceDaoBandanaPersister.java:30)
    at sun.reflect.GeneratedMethodAccessor254.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
    at com.sun.proxy.$Proxy36.retrieve(Unknown Source)
    at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.lambda$retrieve$0(ConfluenceCachingBandanaPersister.java:80)
    at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.lambda$get$11(SynchronousExternalCache.java:239)
    at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.lambda$foldResult$0(CompletionStageUtils.java:19)
    at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
    at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)
    at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)
    at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)
    at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)
    at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.get(SynchronousExternalCache.java:236)
    at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:78)
    at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:71)
    at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
    at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)
    at com.atlassian.confluence.schedule.persistence.dao.CachedScheduledJobDao.getScheduledJobHistory(CachedScheduledJobDao.java:124)
    at com.atlassian.confluence.schedule.managers.DefaultScheduledJobStatusManager.getScheduledJobStatus(DefaultScheduledJobStatusManager.java:52)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener.jobToBeExecuted(ScheduledJobsListener.java:173)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener.lambda$jobToBeExecuted$2(ScheduledJobsListener.java:100)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener$1.doInTransactionWithoutResult(ScheduledJobsListener.java:165)
    at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:34)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener.executeInTransaction(ScheduledJobsListener.java:162)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener.executeInTransactionAndHandleExceptions(ScheduledJobsListener.java:139)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener.jobToBeExecuted(ScheduledJobsListener.java:97)
    at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.runJob(JobRunnerWrapper.java:81)
    at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)
    at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)
    at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:410)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:388)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
    at java.lang.Thread.run(Thread.java:748)
2018-06-04 12:54:37,630 WARN [Caesium-1-2] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
  ->[null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT (Session #1639428673)
2018-06-04 12:54:37,637 ERROR [Caesium-1-2] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob jobToBeExecuted failed to run
org.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only
2018-06-04 12:54:37,648 WARN [Caesium-1-2] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
  ->[null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT (Session #1252186397)
2018-06-04 12:54:37,649 ERROR [Caesium-1-2] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob jobWasExecuted failed to run
org.springframework.orm.hibernate5.HibernateJdbcException: JDBC exception on Hibernate data access:
SQLException for SQL [select confluence0_.BANDANAID as BANDANAID1_7_, confluence0_.BANDANACONTEXT as BANDANACONTEXT2_7_, confluence0_.BANDANAKEY as BANDANAKEY3_7_, confluence0_.BANDANAVALUE
as BANDANAVALUE4_7_ from BANDANA confluence0_ where confluence0_.BANDANACONTEXT=? and confluence0_.BANDANAKEY=?]; SQL state [72000]; error code [1555]; could not execute query;
nested exception is org.hibernate.exception.GenericJDBCException: could not execute query

Thanks

Sanjeev Kumar N

[Sanjeev kumar N]

Hi Thomas,

Was able to find some ERROR in atlassian-confluence.log file, looks like oracle issue.

Could you please suggest me on it.

2018-06-04 12:54:37,614 ERROR [Caesium-1-2] [engine.jdbc.spi.SqlExceptionHelper] logExceptions ORA-01555: snapshot too old: rollback segment number  with name "" too small
ORA-22924: snapshot too old

2018-06-04 12:54:37,615 WARN [Caesium-1-2] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister':
Failed due to UNCLASSIFIED_FAILURE
2018-06-04 12:54:37,617 WARN [Caesium-1-2] [confluence.setup.bandana.ConfluenceDaoBandanaPersister] getObjectFromValue Configuration could not be loaded because class could not be found
 (context: null, key: null).
java.lang.NullPointerException: null
java.lang.NullPointerException
    at java.io.StringReader.<init>(StringReader.java:50)
    at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.getObjectFromValue(ConfluenceDaoBandanaPersister.java:100)
    at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.retrieve(ConfluenceDaoBandanaPersister.java:30)
    at sun.reflect.GeneratedMethodAccessor254.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
    at com.sun.proxy.$Proxy36.retrieve(Unknown Source)
    at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.lambda$retrieve$0(ConfluenceCachingBandanaPersister.java:80)
    at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.lambda$get$11(SynchronousExternalCache.java:239)
    at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.lambda$foldResult$0(CompletionStageUtils.java:19)
    at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
    at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)
    at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)
    at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)
    at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)
    at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.get(SynchronousExternalCache.java:236)
    at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:78)
    at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:71)
    at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
    at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)
    at com.atlassian.confluence.schedule.persistence.dao.CachedScheduledJobDao.getScheduledJobHistory(CachedScheduledJobDao.java:124)
    at com.atlassian.confluence.schedule.managers.DefaultScheduledJobStatusManager.getScheduledJobStatus(DefaultScheduledJobStatusManager.java:52)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener.jobToBeExecuted(ScheduledJobsListener.java:173)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener.lambda$jobToBeExecuted$2(ScheduledJobsListener.java:100)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener$1.doInTransactionWithoutResult(ScheduledJobsListener.java:165)
    at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:34)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener.executeInTransaction(ScheduledJobsListener.java:162)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener.executeInTransactionAndHandleExceptions(ScheduledJobsListener.java:139)
    at com.atlassian.confluence.schedule.listeners.ScheduledJobsListener.jobToBeExecuted(ScheduledJobsListener.java:97)
    at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.runJob(JobRunnerWrapper.java:81)
    at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)
    at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)
    at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:410)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:388)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
    at java.lang.Thread.run(Thread.java:748)
2018-06-04 12:54:37,630 WARN [Caesium-1-2] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
  ->[null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT (Session #1639428673)
2018-06-04 12:54:37,637 ERROR [Caesium-1-2] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob jobToBeExecuted failed to run
org.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only
2018-06-04 12:54:37,648 WARN [Caesium-1-2] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
  ->[null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT (Session #1252186397)
2018-06-04 12:54:37,649 ERROR [Caesium-1-2] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob jobWasExecuted failed to run
org.springframework.orm.hibernate5.HibernateJdbcException: JDBC exception on Hibernate data access:
SQLException for SQL [select confluence0_.BANDANAID as BANDANAID1_7_, confluence0_.BANDANACONTEXT as BANDANACONTEXT2_7_, confluence0_.BANDANAKEY as BANDANAKEY3_7_, confluence0_.BANDANAVALUE
as BANDANAVALUE4_7_ from BANDANA confluence0_ where confluence0_.BANDANACONTEXT=? and confluence0_.BANDANAKEY=?]; SQL state [72000]; error code [1555]; could not execute query;
nested exception is org.hibernate.exception.GenericJDBCException: could not execute query

Thanks

Sanjeev Kumar N

[Moses Thomas]

@LokanaSM

Please try  the resolution  here:

https://confluence.atlassian.com/confkb/confluence-fails-with-ora-01555-error-300812340.html?_ga=2.28126076.1108600623.1528029030-887549751.1487154871

Best!

[LokanaSM]

Thomas,

Thanks for solution, we were able to resolve the issue. Confluence is now accessible. :)

Issues was related to DB, as per the doc DB admin were able to resolve.

Thanks

Sanjeev Kumar N

[Moses Thomas]

@LokanaSMi am happy :) please accept/vote  solution  so  that  others could use solution.

Best!

[LokanaSM]

Thomas,

can i know the steps to accept/vote.

[Moses Thomas]

@LokanaSMCheck  your question, there is accept icon/  vote  answer button on  answer

Best.