[Confluence Data Center 7.19.16] Cannot start server after upgrade Confluence from 7.17.4 to 7.19.16

[Yong Xin Choon]

Hi Community,

We just upgraded our confluence data center server from 7.17.4 to 7.19.16.

The installation process is all normal, no errors.

After that, we checked:

1. mysql-drvier at : /opt/atlassian/confluence/confluence/WEB-INF/lib - it's OK

2. catalina [server.xml] at : /opt/atlassian/confluence/conf/server.xml - it's OK

3. confluence config file [confluence.cfg.xml] at: /var/atlassian/application-data/confluence - it's OK

4. confluence startup env setting file [setenv.sh] at : /opt/atlassian/confluence/bin/setenv.sh - it's OK

Then we tried to start the confluence server by using  [startup.sh]

and we found the catalina.out logs:

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[0.000s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2023-12-03_17-18-39.log instead.
03-Dec-2023 17:18:40.044 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
03-Dec-2023 17:18:40.061 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [368] milliseconds
03-Dec-2023 17:18:40.067 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]
03-Dec-2023 17:18:40.067 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.82]
03-Dec-2023 17:18:41.099 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
03-Dec-2023 17:18:41.234 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath
03-Dec-2023 17:18:41.346 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'
17:18:41 [Catalina-utility-2] o.a.c.c.C.[.[.[/synchrony-proxy][INFO] log - Initializing Spring DispatcherServlet 'dispatcher'
2023-12-03 17:18:41,486 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.19.16 [build 8804 based on commit hash 79a3a8ba7300471318399908b4ee8dff8dc7af4d] - synchrony version 5.1.6
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender (file:/opt/atlassian/confluence/confluence/WEB-INF/lib/org.apache.felix.framework-5.6.12.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

no errors and we checked other logs in : 

/opt/atlassian/confluence/logs

The only abnormal logs we found was :

/var/atlassian/application-data/confluence/logs/atlassian-confluence.log

2023-12-03 17:20:07,801 WARN [Catalina-utility-1] [zaxxer.hikari.pool.ProxyConnection] checkException HikariPool-1 - Connection com.mysql.jdbc.JDBC4Connection@70bfe15f marked as broken because of SQLSTATE(08003), ErrorCode(0)
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:403)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)

2023-12-03 17:20:07,833 ERROR [Catalina-utility-1] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextInitialized Error initialising plugin manager: Could not roll back Hibernate transaction; nested exception is net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection
org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection

...

---

We checked the server port:

Checked the NG network:

not reachable, but before we use the 7.17.4 version, all things are fine.

After we added the heap mem to 12G, we tried to start the confluence server again, and we got this error logs: /var/atlassian/application-data/confluence/logs/atlassian-confluence.log

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'journalEdgeIndexTaskQueue': Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'defaultEdgeIndexTaskFactory': Unsatisfied dependency expressed through constructor parameter 4; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'edgeSearchIndexAccessor': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.confluence.plugins.edgeindex.EdgeSearchIndexAccessor]: Constructor threw exception; nested exception is com.atlassian.bonnie.LuceneException: java.nio.file.NoSuchFileException: /var/atlassian/application-data/confluence/index/edge/_ot8.si

Caused by: java.nio.file.NoSuchFileException: /var/atlassian/application-data/confluence/index/edge/_ot8.si
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:223)
at org.apache.lucene.codecs.lucene40.Lucene40SegmentInfoReader.read(Lucene40SegmentInfoReader.java:50)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:334)
at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:380)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:812)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:663)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:383)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:711)
at com.atlassian.bonnie.LuceneConnection.upgradeIndexIfNecessary(LuceneConnection.java:608)
at com.atlassian.bonnie.LuceneConnection.ensureCorrectIndexFormat(LuceneConnection.java:593)
at com.atlassian.bonnie.LuceneConnection.setUpWriterAndSearcher(LuceneConnection.java:557)
... 89 more

Anyone know what is this '_ot8.si' file ？

Thx/Br

[Tinker Fadoua]

@Yong Xin Choon 

did you check if the user account has all required access to the files on the server?

Also even though it is showing as a WARN but there is some problem with DB

Please keep us posted 

Best,

Fadoua

[Tinker Fadoua]

Also did you disable all plugins before upgrade?

[Yong Xin Choon]

Hi Fadoua,

Thanks for your reply, at last we started the Confluence server by adding the heap mem to 12G, but the error still there : 

java.nio.file.NoSuchFileException: /var/atlassian/application-data/confluence/index/edge/_ot8.si

For your questions:

"did you check if the user account has all required access to the files on the server?"

answer is : yes, the user account has all required access to the files on the server.

"Also did you disable all plugins before upgrade?"

answer is : no, we just run the installer directly.

Thx/Br