Confluence Error

[atsflex]

We are having issue(s) upgrading to the latest version of confluence.  We are going from 5.7.1 to 6.5.  But on the test server we're getting the following error:

SEVERE: The web application [] appears to have started a thread named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] but has failed to stop it. This is very likely to create a memory leak.
Nov 30, 2017 12:49:37 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1] but has failed to stop it. This is very likely to create a memory leak.
Nov 30, 2017 12:49:37 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2] but has failed to stop it. This is very likely to create a memory leak.
Nov 30, 2017 12:49:37 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8090"]
Nov 30, 2017 12:49:37 PM org.apache.catalina.startup.Catalina start

Has anyone seen this error before? Any help/feedback is welcome

[Nic Brough -Adaptavist-]

Ohhh, I think I know this one, but rather than go into lots of potentially useless essay-like answers, I'll ask a question to see if my guess is right.

What database are you using for the 5.7 install?  Is it the internal HSQL one?

[atsflex]

We are using Oracle 12.10.  We  have 5.7.1 running in our Production environment successful but having this issue in our test environment.   We need 5.7.1 running in our test environment successful, then upgrade to the latest version (6.5).  Then, upgrade to 6.5 in Production Environment.  Thanks for your guidance and help on this issue.   

[atsflex]

Any updates or ideas on how to proceed? 

[AnnWorley]

Nik, please let us know what steps you took to upgrade - whether you used an archive file or an installer, etc.

The shutdown errors you posted usually appear after some event that caused Confluence to stop running. To provide an example of what a healthy upgrade log looks like, I ran one on a  test instance. 

To figure out what went wrong, look for the log to proceed as in this example, and focus on where it diverges - especially if you see ERROR messages or the phrase "upgrade failed application will not start"

When you launch the upgrade you should see the upgraded version starting:

2017-12-05 14:11:12,427 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.5.0 [build 7502 based on commit hash b9217199c27dd617c8d51f6186067d7767c5001b] - synchrony version 2.1.0-release-confluence_6.5-8d720dbe

Next there will be lines and lines of alter database entries:

2017-12-05 14:12:11,808 INFO [localhost-startStop-1] [confluence.upgrade.impl.DefaultUpgradeManager] beforeUpgrade Finished generating pre-upgrade recovery file.
2017-12-05 14:12:15,367 INFO [localhost-startStop-1] [confluence.upgrade.upgradetask.Hibernate5ConstraintNamesPreSchemaUpgradeTask] doUpgrade Starting Hibernate5ConstraintNamesPreSchemaUpgradeTask
2017-12-05 14:12:15,368 INFO [localhost-startStop-1] [confluence.upgrade.ddl.HibernateAlterTableExecutor] alterTable Executing grouped alter table command on ATTACHMENTDATA
2017-12-05 14:12:15,392 INFO [localhost-startStop-1] [confluence.upgrade.ddl.HibernateDdlExecutor] executeDdlStatements Executing DDL: alter table ATTACHMENTDATA DROP CONSTRAINT FK9DC3E34D34A4917E
2017-12-05 14:12:15,439 INFO [localhost-startStop-1] [confluence.upgrade.ddl.HibernateAlterTableExecutor] alterTable Executing grouped alter table command on BODYCONTENT
2017-12-05 14:12:15,439 INFO [localhost-startStop-1] [confluence.upgrade.ddl.HibernateDdlExecutor] executeDdlStatements Executing DDL: alter table BODYCONTENT DROP CONSTRAINT FKA898D4778DD41734
2017-12-05 14:12:15,458 INFO [localhost-startStop-1] [confluence.upgrade.ddl.HibernateAlterTableExecutor] alterTable Executing grouped alter table command on CONFANCESTORS
2017-12-05 14:12:15,459 INFO [localhost-startStop-1] [confluence.upgrade.ddl.HibernateDdlExecutor] executeDdlStatements Executing DDL: alter table CONFANCESTORS DROP CONSTRAINT FK9494E23C37E35A2E
2017-12-05 14:12:15,462 INFO [localhost-startStop-1] [confluence.upgrade.ddl.HibernateAlterTableExecutor] alterTable Executing grouped alter table command on CONFANCESTORS
2017-12-05 14:12:15,463 INFO [localhost-startStop-1] [confluence.upgrade.ddl.HibernateDdlExecutor] executeDdlStatements Executing DDL: alter table CONFANCESTORS DROP CONSTRAINT FK9494E23CC45E94DC
2017-12-05 14:12:15,465 INFO [localhost-startStop-1] [confluence.upgrade.ddl.HibernateAlterTableExecutor] alterTable Executing grouped alter table command on CONTENT

Finally:

2017-12-05 14:12:35,024 INFO [localhost-startStop-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] entireUpgradeFinished Upgrade completed successfully

and after a list of plugins loading:

2017-12-05 14:12:37,132 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve

If your upgrade is not getting that far there should be an error message with a cause.

[atsflex]

I am proceed with the 6.5.0 installation vs hitting my head on 5.7.1.  I think there were some issues with the application server with 5.7.1 so I am going to try 6.5.0 and try to mirror this for production.  

[AnnWorley]

Hi Nik,

Those "SEVERE" memory leak errors in catalina.out are common and benign.

The upgrade tasks are written to Confluence_Home/logs/atlassian-confluence.log. Please monitor that log as the upgrade proceeds. If there is a problem the best clues will be found there.

If you aborted the upgrade after the SEVERE errors, please restore the home directory and database to the pre-upgrade state and launch the upgrade again. Please watch the atlassian-confluence log from the line "Starting Confluence 6.5" to the line "Confluence is ready to serve." There will be a lot of entries indicating database updates.

At that point Confluence will be available in the browser. If you see an entry "upgrade failed..." there should be a reason in the error message, so please post it here if we can be of help.

Thanks,

Ann

[atsflex]

In atlassian-confluence.log I received the following error. 

[root@jira103 logs]# tail atlassian-confluence.log
        at net.sf.hibernate.dialect.Dialect.getSequenceNextValString(Dialect.java:335)
        at net.sf.hibernate.id.SequenceGenerator.configure(SequenceGenerator.java:57)
        at net.sf.hibernate.id.IdentifierGeneratorFactory.create(IdentifierGeneratorFactory.java:78)
        ... 68 more
2017-12-04 08:48:35,391 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] shutdown Preparing to shut down the plugin system
2017-12-04 08:48:35,392 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] shutdown Shutting down the plugin system
2017-12-04 08:48:35,397 WARN [localhost-startStop-1] [atlassian.plugin.jmx.JmxUtil] unregister Failed to unregister, instance not found:
2017-12-04 08:48:35,464 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] destroy Shutting down long running task service
2017-12-04 08:48:35,610 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] shutdownCacheManager Shutting down EhCache cache manager
2017-12-04 08:48:35,625 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextDestroyed Stopping Confluence

[atsflex]

The web application [] appears to have started a thread named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1] but has failed to stop it. This is very likely to create a memory leak

[Gonchik Tsymzhitov]

Hello Nick, 

Looks like you JVM memory and thread pool configuration is small.

Have you reviewed when you upgraded?   

https://confluence.atlassian.com/doc/confluence-6-5-upgrade-notes-938862633.html#Confluence6.5UpgradeNotes-Upgradenotes

Cheers, 

Gonchik Tsymzhitov

[atsflex]

 I reviewed the JVM memory and I already had it set to below.   

CATALINA_OPTS="$CATALINA_OPTS -Xms1024m -Xmx1024m -XX:MaxPermSize=384

In atlassian-confluence.log is the following. 

 at net.sf.hibernate.dialect.Dialect.getSequenceNextValString(Dialect.jav                                    a:335)
        at net.sf.hibernate.id.SequenceGenerator.configure(SequenceGenerator.jav                                    a:57)
        at net.sf.hibernate.id.IdentifierGeneratorFactory.create(IdentifierGener                                    atorFactory.java:78)
        ... 68 more
2017-11-30 14:11:14,427 INFO [localhost-startStop-1] [atlassian.plugin.manager.D                                    efaultPluginManager] shutdown Preparing to shut down the plugin system
2017-11-30 14:11:14,428 INFO [localhost-startStop-1] [atlassian.plugin.manager.D                                    efaultPluginManager] shutdown Shutting down the plugin system
2017-11-30 14:11:14,431 WARN [localhost-startStop-1] [atlassian.plugin.jmx.JmxUt                                    il] unregister Failed to unregister, instance not found:
2017-11-30 14:11:14,506 INFO [localhost-startStop-1] [com.atlassian.confluence.l                                    ifecycle] destroy Shutting down long running task service
2017-11-30 14:11:14,640 INFO [localhost-startStop-1] [com.atlassian.confluence.l                                    ifecycle] shutdownCacheManager Shutting down EhCache cache manager
2017-11-30 14:11:14,668 INFO [localhost-startStop-1] [com.atlassian.confluence.l                                    ifecycle] contextDestroyed Stopping Confluence