Jira fails to start with lots of "Cannot execute atlassian-spring-scanner-runtime" exceptions

[Richard Sinden]

Hi,

I have a Jira 7.6.1 instance running on windows, and most of the time it fails to start and generates lots of the following exceptions for various plugins, then sits there waiting for plugins to finish loading until it times out. Occasionally it doesn't generate these errors and loads the plugins with no problem, but I'm struggling to find a pattern. It seems to fail more often if I start it as a service, rather than from the command line, but it isn't always the case. The log file before this point looks the same whether it is going to succeed or fail.

Any ideas?

2017-12-11 00:11:48,810 JIRA-Bootstrap INFO      [c.a.plugin.manager.DefaultPluginManager] Plugin system earlyStartup begun
2017-12-11 00:12:02,061 ThreadPoolAsyncTaskExecutor::Thread 4 ERROR      [c.a.p.osgi.factory.OsgiPlugin] Unable to start the plugin container for plugin 'com.atlassian.jira.jira-project-config-plugin'
org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from URL [bundle://71.0:0/META-INF/spring/spring-scanner.xml]; nested exception is java.lang.IllegalStateException: Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes, perhaps embedded inside the target plugin 'com.atlassian.jira.admin-project-config-plugin'; embedding scanner-annotations is not supported since scanner version 2.0. Use 'mvn dependency:tree' and ensure the atlassian-spring-scanner-annotation dependency in your plugin has <scope>provided</scope>, not 'runtime' or 'compile', and you have NO dependency on atlassian-spring-scanner-runtime.
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
    at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
    at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
    at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
    at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:170)
    at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:140)
    at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
    at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationContext.java:60)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.java:242)
    at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationContext.java:220)
    at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:224)
    at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:177)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:157)
    at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager$1.run(LifecycleManager.java:207)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes, perhaps embedded inside the target plugin 'com.atlassian.jira.admin-project-config-plugin'; embedding scanner-annotations is not supported since scanner version 2.0. Use 'mvn dependency:tree' and ensure the atlassian-spring-scanner-annotation dependency in your plugin has <scope>provided</scope>, not 'runtime' or 'compile', and you have NO dependency on atlassian-spring-scanner-runtime.
    at com.atlassian.plugin.spring.scanner.runtime.impl.AtlassianScannerBeanDefinitionParser.checkScannerRuntimeIsNotEmbeddedInBundle(AtlassianScannerBeanDefinitionParser.java:198)
    at com.atlassian.plugin.spring.scanner.runtime.impl.AtlassianScannerBeanDefinitionParser.parse(AtlassianScannerBeanDefinitionParser.java:60)
    at org.springframework.beans.factory.xml.NamespaceHandlerSupport.parse(NamespaceHandlerSupport.java:74)
    at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1411)
    at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
    at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
    at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
    ... 20 more

Thanks,

Richard

[Andy Heinzer]

If Jira's plugins are sometimes timing out when trying to start up, it tends to be because the system is under resourced for all the load the plugins in Jira are putting on the JVM.  I would be interested to learn more details about your specific environment, such as JVM Xmx value, the total system memory, the number of CPU cores and their speed, as well as the number of issues in Jira, and details like the plugins being loaded by Jira.   It could also be that other applications running on the same system, like a database instance or other applications like Confluence could be using up resources at the time Jira is trying to start up which might be a factor here.

From there I would want to compare your system against our JIRA Sizing Guide - Atlassian Documentation This is one way to understand if you need to Increasing memory in JIRA.   You could also follow the work-around in the KB JIRA applications System Plugin Timeout While Waiting for Add-ons to Enable.  That KB has a JVM argument you can add to Jira to tell the system to wait longer in order to load all the system plugins necessary to make Jira work.

It's also possible that a system plugin can fail to start up a number of times in Jira and as a result it can get flagged in the database as a marker to disable this plugin.  If that happens then you would need to remove the entry that corresponds to this plugin in the pluginstate table.  You can see what plugins are currently disabled in this Jira instance by running the command:

select * from pluginstate;

If any entries are listed here as system plugins and are disable, then you would need to clear those entries from the table and restart Jira once more to get these system plugins to load on startup.

[Richard Sinden]

Hi Andrew,

Thanks for the reply. It is a tiny system - 5 users, ~300 issues. It is running in a Windows 2016 VM on a 4-core Xeon E3-1225 v3 with 12GBs of RAM shared between a couple of other VMs that aren't really doing anything at all. CPU usage is stable at <10% and RAM is < 50% until Jira startup begins.

Whilst the plugins do timeout when it fails, all the exceptions seem to happen right at the start of the plugin load. When it does succeed then all the plugins load in <20 seconds, so I don't think performance is an issue here.

I tried increasing the Xmx value to 1024, but this didn't help, and neither did increasing the plugin wait timeout as I'm pretty certain they've failed and are never going to start.

I checked the pluginstate table, but it appeared to be empty. Is this what you'd expect?

If you've got any other ideas, I'd be very keen to hear them.

I've got full log files for a failed and successful startup, which I guess might be useful. I tried pasting them into a codeblock here, but I got an error when I hit reply saying it wasn't a valid message. Is there an easy way to share them?

Many thanks,

Richard

[Richard Sinden]

Actually, I thought I'd just stick the logs in pastebin:

1st part of bad log: https://pastebin.com/2Rx26yMZ
2nd part of bad log: https://pastebin.com/uSdWk22B

Good log: https://pastebin.com/30d1Ay1C

The good log contains a list of the plugins.

Thanks,

Richard

[Andy Heinzer]

Hi Richard,

Thanks for these logs. It's very strange that this would sometimes work and other times not. It isn't clear to me exactly what would cause that behavior just yet. Did this problem start just after the most recent upgrade to 7.6.1? Or was this problem in existence before the upgrade?

I did also notice in your logs this warning:

2017-12-10 23:28:44,976 JIRA-Bootstrap WARN      [c.a.jira.health.HealthChecks] Your database is using an unsupported collation
2017-12-10 23:28:44,976 JIRA-Bootstrap WARN      [c.a.jira.health.HealthChecks] Your postgres72 database is currently using an unsupported collation: English_United States.1252. You should change this to a supported collation:  - POSIX.UTF-8
        - C.UTF-8
        - C
        - POSIX
   
    Review our documentation for more information on supported collations.  

I have seen all kinds of strange malfunctions in Jira for platforms using unsupported collations/encoding. It might not be a cause here, but I think it is worth ruling this out by fixing this collation encoding problem with these steps:

1. Select a database type/version from our Supported platforms - Atlassian Documentation
2. Create a new blank database per Connecting Jira applications to PostgreSQL, (paying close attention to the correct collation for that database type)
3. Gather an XML backup of your JIRA instance. Even if you can't start JIRA, you can still find a recent one of these backups in the filesystem of the JIRA server. By default this is saved in <jira-home>/export/ folder
4. Stop JIRA,
5. Run the JIRA Configuration tool to tell JIRA to use the new blank database, save these changes, (if you can't run this, then you can just directly edit the <jira-home>/dbconfig.xml file to make these changes)
6. Start JIRA up again (when JIRA starts with an empty database, it automatically launches the setup wizard)
7. Copy the export XML zip file from <jira-home>/export/ into <jira-home>/import/
8. Then you can import the backup using the 'Import your data' link in the setup wizard

By following these steps you can then import your previous data into a supported database with the correct database setup.

Please try these steps and let me know if you continue to see this kind of startup problem.
Regards,
Andy

[Frank Butzek]

Was there a solution to this? The collation shouldn't be the problem as it is correct in my case. I administer some instances and I faced the exact same problem for the first time now on a Windows Server 2016 - which is my guess for the problem with JIRA (7.7.2 in my case).