Confluence had failed to start after upgrade from 6.4 to 6.5.2

[Fabio Martignago]

hello,

I get this message:

"

Confluence had failed to start, but we're back up and running again

Take me to my Confluence home.

Finishing up checks

"

after upgrade from 6.4 to 6.5.2

What can I do?

[Fabio Martignago]

I've upgraded the SQL Server from 2008R2 to 2014 without any issues.

Now I'm using SQL Server 2014. but the same errors remains.

from Event Viewer: Event ID 824

SQL Server detected a logical consistency-based I/O error: incorrect pageid (expected 1:22500; actual 0:0). It occurred during a read of page (1:22500) in database ID 5 at offset 0x0000000afc8000 in file 'C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\Confluence.mdf'.  Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

I could try to export the entire Confluence configuration, shutdown this Confluence database and then create a new one where to restore it.

Sounds good?

[AnnWorley]

It sounds good if we are on the same page:

1. Create a new database according to Database Setup for SQL Server
2. Install Confluence using that database (no data)
3. Restore an XML site backup from the instance that is having the consistency issues.

So you know what to expect from the site restore, please review: Restoring a Site

If you have a different plan, please let me know, my mind is open. :)

I am not a SQL server expert so maybe you can enlighten me - I am curious what you would expect to happen if you ran the database consistency check indicated in the error message.

In some cases, for example, if unique or foreign key constraints are violated by data inconsistency, the XML backup won't restore. Please update this thread to let me know if you run into such issues.

[Fabio Martignago]

hello @AnnWorley, I found this bad error (in both environment: production 6.4 and demo 6.6.x)

SQL Server detected a logical consistency-based I/O error: incorrect pageid (expected 1:22500; actual 0:0). It occurred during a read of page (1:22500) in database ID 5 at offset 0x0000000afc8000 in file 'C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\Confluence.mdf'. Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

[AnnWorley]

Hi Fabio,

The file path in the error (C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA) is a default install path for SQL Server 2008. The supported platforms for both Confluence 6.4 and 6.6.x include SQL Server 2012 and 2014 but not 2008. Please verify you are using a supported version of MS SQL Server, as version differences could cause database consistency issues.

Thanks,

Ann

[Fabio Martignago]

in the log, last ERROR entry:

2018-01-17 09:33:41,552 ERROR [localhost-startStop-1] [atlassian.config.lifecycle.LifecycleManager] panicAndShutdown Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:mananagedjobs (Start and stop the Managed Scheduled Jobs)
org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection
at org.springframework.orm.hibernate.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:532)

....

Caused by: net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection

[AnnWorley]

I have seen that error "Unable to rollback against JDBC Connection" on startup related to Oracle database drivers (Unable to start up Confluence. Fatal error during startup sequence) but you are using MS SQL. 

The MS SQL driver was changed in Confluence 6.6:  SQL Server driver update

To help isolate the issue, please check which driver is specified for your instance in the file: <Confluence_home>/confluence.cfg.xml. Please look for: 

<property name="hibernate.connection.driver_class"

I look forward to hearing whether the upgrade successfully changed your MS SQL driver or if it is running with an outdated jtds driver, perhaps.

[Fabio Martignago]

hello,

here again. 

After I took a VM snap , I tried to upgrade the latest version 6.6.1 but the same, I get this upgrade error: (more detailed than the past 6.5.x update):

Confluence had problems starting up
This page is for Confluence administrators. If you're seeing this page, your Confluence administrator is probably working to restore the service.

Description Time Level Exception
Upgrade failed. Please consult the system logs for details. You will need to fix these problems, restore your database and confluence home directory to the pre upgrade state. Then retry the upgrade. Upgrade error message: Cannot update schema
2018-01-10 16:14:08 error
Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:mananagedjobs (Start and stop the Managed Scheduled Jobs)
2018-01-10 16:15:49 fatal Could not roll back Hibernate transaction; nested exception is net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection

[AnnWorley]

Sometimes the database related error messages in the logs are really detailed and helpful.

Please check your <confluence_home>/logs/atlassian-confluence.log logs for errors like "Cannot update schema". If you find a long ERROR stack trace, search for a "caused by:" line and please let us know what you find.

[Fabio Martignago]

ok, I reproduce the error to another test environment, and this is the last ERROR entry:

 2018-01-16 16:02:23,684 ERROR [localhost-startStop-1] [atlassian.config.lifecycle.LifecycleManager] panicAndShutdown Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:mananagedjobs (Start and stop the Managed Scheduled Jobs)
org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection
 at org.springframework.orm.hibernate.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:532)
 at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doRollback(ConfluenceHibernateTransactionManager.java:81)
 at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:853)
 at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:830)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
 at com.sun.proxy.$Proxy53.rollback(Unknown Source)
 at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:503)
 at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:285)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
 at com.sun.proxy.$Proxy159.initialiseTriggers(Unknown Source)
 at com.atlassian.confluence.schedule.listeners.ManagedScheduledJobsLifecycle.startup(ManagedScheduledJobsLifecycle.java:24)
 at com.atlassian.confluence.impl.setup.DefaultLifecycleManager.startUp(DefaultLifecycleManager.java:49)
 at com.atlassian.config.lifecycle.LifecycleServletContextListener.contextInitialized(LifecycleServletContextListener.java:17)
 at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4852)
 at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5313)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection
 at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:119)
 at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.rollback(JdbcResourceLocalTransactionCoordinatorImpl.java:239)
 at org.hibernate.engine.transaction.internal.TransactionImpl.rollback(TransactionImpl.java:100)
 at com.atlassian.hibernate.adapter.adapters.TransactionV2Adapter.rollback(TransactionV2Adapter.java:41)
 at org.springframework.orm.hibernate.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:529)
 ... 28 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
 at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:229)
 at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:992)
 at com.microsoft.sqlserver.jdbc.SQLServerConnection.rollback(SQLServerConnection.java:2908)
 at com.mchange.v2.c3p0.impl.NewProxyConnection.rollback(NewProxyConnection.java:1033)
 at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:114)
 ... 32 more

[AnnWorley]

I think you may have found our culprit:

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.

Please see if there is a validation query set for the database as described in: Surviving Database Connection Closures

The validation query is there when you do a new install of Confluence 6.5 or above but if you upgrade from pre-6.5 (like Confluence 6.4) you can be vulnerable to the issue described in the link above.

It seems like we are making progress, I hope the validation query enables the upgrade to complete.

[Fabio Martignago]

thx @AnnWorley, I did the first point: "Enable validation query with a direct JDBC connection" but I got the same error

Upgrade failed. Please consult the system logs for details. You will need to fix these problems, restore your database and confluence home directory to the pre upgrade state. Then retry the upgrade. Upgrade error message: Cannot update schema

 Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:mananagedjobs (Start and stop the Managed Scheduled Jobs)

[AnnWorley]

The message sounds encouraging, "we're back up and running again". what happens when you click on "Take me to my Confluence home."?

If the upgrade failed there should be an error message to that effect in the <Confluence_Home>/logs/atlassian-confluence.log.

I look forward to hearing more about the upgrade - is it a Production or test instance? Did you use an installer or archive file to upgrade?

[Fabio Martignago]

thx Ann,
when I click on it, I move to the same error page.
I used the atlassian-confluence-6.5.2-x64.exe installer.

Because it's a production site I was little bit scared so I resumed a full image of the entire virtual machine.
Now it's running the previus 6.4 version.

I would like to upgrade again, but being aware that I suppose I will have the same issue.

[AnnWorley]

I recommend trying the upgrade on a test instance first; here is a doc on migrating to another server so you can use it as a test instance: Migrating Confluence Between Servers

Please review the upgrade guide before the next time so you will have the latest procedure at that time: Upgrading Confluence 

You may be interested in Confluence 6.6: Confluence 6.6 Upgrade Notes so I thought I would mention that it has been released.