Upgraded 7.7.3 to 7.13.0 - confluence doesn't start

[tgix-msa]

Just updated our installation from 7.7.3 to 7.13.0 and confluence no longer completes the startup procedure.

After waiting several minutes, the java process stops consuming CPU and just sits idle. I have waited 15 minutes, but still not possible to login or any new messages in log. I have tried with the stock server.xml as well.

Running on debian with mysql and java 1.8.0_121 (same as 7.7.3 was running fine on).

Where should I start looking for a solution?

catalina.out after starting 7.13.0:

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[0.000s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/home/confluence/atlassian/confluence/logs/gc-2021-08-26_11-22-09.log instead.
26-Aug-2021 11:22:09.597 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server] failed to set property [debug] to [0]
26-Aug-2021 11:22:09.722 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]
26-Aug-2021 11:22:09.757 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine] failed to set property [debug] to [0]
26-Aug-2021 11:22:09.761 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [debug] to [0]
26-Aug-2021 11:22:09.839 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]
26-Aug-2021 11:22:09.885 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]
26-Aug-2021 11:22:10.510 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
26-Aug-2021 11:22:10.566 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1144] milliseconds
26-Aug-2021 11:22:10.575 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]
26-Aug-2021 11:22:10.577 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.45]
26-Aug-2021 11:22:13.163 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
log4j:ERROR Could not find value for key log4j.appender.luceneQuery
log4j:ERROR Could not instantiate appender named "luceneQuery".
26-Aug-2021 11:22:13.540 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath
26-Aug-2021 11:22:13.966 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'
11:22:13 [Catalina-utility-2] o.a.c.c.C.[.[.[/synchrony-proxy][INFO] log - Initializing Spring DispatcherServlet 'dispatcher'
2021-08-26 11:22:14,561 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.13.0 [build 8703 based on commit hash 791df1c41ad1e432a490f9fc6f60e3ea49358bee] - synchrony version 4.0.0-master-f680ea21
2021-08-26 11:22:19,803 WARN [Catalina-utility-1] [confluence.impl.setup.DefaultBootstrapDatabaseAccessor] readValue Unable to determine ZDU status from database. If you are upgrading from a Confluence version prior to 7.9, this is expected.
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'confluencedb.CONFZDU' doesn't exist
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:943)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3970)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3906)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2677)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2503)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1369)
at com.atlassian.confluence.impl.setup.DefaultBootstrapDatabaseAccessor.readValue(DefaultBootstrapDatabaseAccessor.java:79)
at com.atlassian.confluence.impl.setup.DefaultBootstrapDatabaseAccessor.getBootstrapData(DefaultBootstrapDatabaseAccessor.java:48)
at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:805)
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:69)
at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:191)
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:144)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:63)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5172)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.atlassian.hibernate.adapter.proxy.BytecodeProviderImpl_ImplementV2Proxy (file:/home/confluence/atlassian/confluence/confluence/WEB-INF/lib/hibernate.adapter-1.0.3.jar) to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of com.atlassian.hibernate.adapter.proxy.BytecodeProviderImpl_ImplementV2Proxy
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Security framework of XStream not explicitly initialized, using predefined black list on your own risk.

[tgix-msa · Accepted solution]

Let it sit overnight and now found the instance at least started:

27-Aug-2021 03:26:37.565 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
27-Aug-2021 03:26:37.665 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [57867097] milliseconds
03:26:38 [http-nio-8090-exec-11] o.e.j.u.log[INFO] initialized - Logging initialized @57869906ms to org.eclipse.jetty.util.log.Slf4jLog

Errors logged on the webpage when trying to access:

Pre-Upgrade recovery file generation failed.
Please refer to https://confluence.atlassian.com/x/ropKGQ for possible solution.

I will try the instructions there and disable backup file generation. Disk space is not an issue.

[tgix-msa]

Got everything back up again. 

[Martin Henning]

Letting it sit worked. I left mine for about an hour and it came up