Confluence dies sporadically after boot on Debian 8

[Frank Schmidt]

Hi,

I upgraded a copy of our production server from version 5.8.9 to version 6.3.1 in a staging environment.

Now Confluence is dying sporadically shortly after the boot process has finished for no apparent reason. This happens about every 5th to 10th boot. The only errors in <confluence_home>/logs/atlassian-confluence.log are:

2017-07-24 14:10:14,646 ERROR [ThreadPoolAsyncTaskExecutor::Thread 13] [spring.scanner.extension.ComponentImportBeanFactoryPostProcessor] postProcessBeanFactory Unable to load class 'com.atlassian.bitbucket.auth.RememberMeService' for component importation purposes.  Skipping...
2017-07-24 14:10:14,652 ERROR [ThreadPoolAsyncTaskExecutor::Thread 13] [spring.scanner.extension.ComponentImportBeanFactoryPostProcessor] postProcessBeanFactory Unable to load class 'com.atlassian.jira.config.properties.ApplicationProperties' for component importation purposes.  Skipping...

As these error messages can also be seen during successful Confluence startups I suppose that they are not the cause of the problem.

According to /var/log/syslog the service is started normally:

Jul 24 14:09:19 confluencetest confluence[1041]: To run Confluence in the foreground, start the server with start-confluence.sh -fg
Jul 24 14:09:19 confluencetest confluence[1041]: executing using dedicated user: confluence
Jul 24 14:09:19 confluencetest confluence[1041]: If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide
Jul 24 14:09:19 confluencetest confluence[1041]: Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
Jul 24 14:09:19 confluencetest confluence[1041]: ---------------------------------------------------------------------------
Jul 24 14:09:19 confluencetest confluence[1041]: Using Java: /opt/atlassian/confluence/jre//bin/java
Jul 24 14:09:19 confluencetest systemd[1]: Started MySQL Community Server.
Jul 24 14:09:20 confluencetest confluence[1041]: 2017-07-24 14:09:20,756 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
Jul 24 14:09:20 confluencetest confluence[1041]: ---------------------------------------------------------------------------
Jul 24 14:09:20 confluencetest confluence[1041]: Using CATALINA_BASE:   /opt/atlassian/confluence
Jul 24 14:09:20 confluencetest confluence[1041]: Using CATALINA_HOME:   /opt/atlassian/confluence
Jul 24 14:09:20 confluencetest confluence[1041]: Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp
Jul 24 14:09:20 confluencetest confluence[1041]: Using JRE_HOME:        /opt/atlassian/confluence/jre/
Jul 24 14:09:20 confluencetest confluence[1041]: Using CLASSPATH:       /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar
Jul 24 14:09:20 confluencrtest confluence[1041]: Using CATALINA_PID:    /opt/atlassian/confluence/work/catalina.pid
Jul 24 14:09:20 confluencetest confluence[1041]: Tomcat started.

Has anyone experienced a similar behaviour of a Confluence server?

As I could not find a solution to this behaviour after investigating the problem for a longer period of time and searching the Atlassian documentation, knowledgebase etc. any help on this is appreciated!

[GÉANT IT]

"Me too".

I'm experiencing the same thing on Ubuntu 16.04.

I run several instances and today I saw this behavior on two instances, one running 6.1.3, and on 6.3.1.

It's weird because it mostly seems to happen during unattended reboots. Manual reboots look to be OK - but I don't have hard numbers.

Slightly worrying that there is no consistency in these.

[Frank Schmidt]

Update:

Using a fresh Installation of Confluence and Debian 8 (using same versions as above, no user generated data and no additional plugins) I get the same behavour:

Confluence dies again from time to time a few seconds after the boot process has finished.

<confluence_home>/logs/atlassian-confluence.log is similiar to my first post:

2017-07-25 14:44:28,470 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.3.1 [build 7202 based on commit hash 57e26a1771e14e3fb96fd7cebe99a87f17d4379c] - synchrony version 1.0.0-release-confluence_6.1-a1ab321e
2017-07-25 14:44:30,650 INFO [localhost-startStop-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$populateExistingClusterSetupConfig$1 Populating setup configuration if running with Cluster mode...
2017-07-25 14:44:30,744 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2017-07-25 14:44:34,167 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager
2017-07-25 14:44:43,932 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization completed in 13188 ms
2017-07-25 14:44:45,070 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] earlyStartup Plugin system earlyStartup begun
2017-07-25 14:44:59,488 WARN [ThreadPoolAsyncTaskExecutor::Thread 19] [spring.scanner.util.ProductFilterUtil] detectProduct Couldn't detect product, will use ProductFilter.ALL
2017-07-25 14:44:59,573 WARN [ThreadPoolAsyncTaskExecutor::Thread 1] [spring.scanner.util.ProductFilterUtil] detectProduct Couldn't detect product, will use ProductFilter.ALL
2017-07-25 14:45:00,646 WARN [ThreadPoolAsyncTaskExecutor::Thread 12] [spring.scanner.util.ProductFilterUtil] detectProduct Couldn't detect product, will use ProductFilter.ALL
2017-07-25 14:45:12,088 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system earlyStartup ended
2017-07-25 14:45:12,113 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] lateStartup Plugin system lateStartup begun
2017-07-25 14:45:14,501 ERROR [ThreadPoolAsyncTaskExecutor::Thread 18] [spring.scanner.extension.ComponentImportBeanFactoryPostProcessor] postProcessBeanFactory Unable to load class 'com.atlassian.bitbucket.auth.RememberMeService' for component importation purposes.  Skipping...
2017-07-25 14:45:14,501 ERROR [ThreadPoolAsyncTaskExecutor::Thread 18] [spring.scanner.extension.ComponentImportBeanFactoryPostProcessor] postProcessBeanFactory Unable to load class 'com.atlassian.jira.config.properties.ApplicationProperties' for component importation purposes.  Skipping...
2017-07-25 14:45:14,549 INFO [ThreadPoolAsyncTaskExecutor::Thread 11] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2017-07-25 14:45:14,762 WARN [ThreadPoolAsyncTaskExecutor::Thread 19] [client.eventfilter.whitelist.AnalyticsWhitelistModuleDescriptor] readPluginWhitelist No whitelist resource file found at [mediaviewer/lib/analytics-whitelist.json] in plugin [com.atlassian.confluence.plugins.confluence-previews] (bundle entry not found)
2017-07-25 14:45:14,762 WARN [ThreadPoolAsyncTaskExecutor::Thread 19] [client.eventfilter.whitelist.AnalyticsWhitelistModuleDescriptor] readPluginWhitelist No whitelist resource file found at [mediaviewer/lib/analytics-whitelist.json] in plugin [com.atlassian.confluence.plugins.confluence-previews] (null InputStream)
2017-07-25 14:45:16,870 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system lateStartup ended
2017-07-25 14:45:16,964 INFO [lifecycle:thread-5] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2017-07-25 14:45:16,966 WARN [lifecycle:thread-20] [expose.jmx.schedule.JmxInstrumentSchedulerImpl] onStart atlassian-instrumentation-jmx expose scheduler started.
2017-07-25 14:45:17,032 WARN [lifecycle:thread-13] [hql.internal.ast.HqlSqlWalker] generatePositionalParameter [DEPRECATION] Encountered positional parameter near line 1, column 93 in HQL: [FROM com.atlassian.confluence.impl.schedule.caesium.SchedulerClusteredJob t WHERE t.jobId = ?].  Positional parameter are considered deprecated; use named parameters or JPA-style positional parameters instead.