Confluence: BeanDefinitionStoreException: does not load Plugins due to Spring-Scanner Exceptions

[Matthias Steinböck]

Our client upgraded from Confluence 6.4.1 to 6.7.2

Sometimes we get BeanDefinitionStoreExceptions on startup leaving some plugins disabled:

2018-03-24 12:54:14,794 ERROR [ThreadPoolAsyncTaskExecutor::Thread 18] [internal.dependencies.startup.DependencyWaiterApplicationContextExecutor] fail Unable to create application context for [at.example.confluence.exampleplugin.example-plugin], unsatisfied dependencies: none
org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from URL [bundle://230.0:0/META-INF/spring/plugin-context.xml]; nested exception is java.lang.IllegalStateException: Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes, perhaps embedded inside the target plugin 'at.example.confluence.exampleplugin.example-plugin'; embedding scanner-annotations is not supported since scanner version 2.0. Use 'mvn dependency:tree' and ensure the atlassian-spring-scanner-annotation dependency in your plugin has <scope>provided</scope>, not 'runtime' or 'compile', and you have NO dependency on atlassian-spring-scanner-runtime.
 at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
 at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
 at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
 at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
 at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
 at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
 at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:170)
 at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:140)
 at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
 at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
 at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationContext.java:60)
 at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.java:242)
 at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
 at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationContext.java:220)
 at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:224)
 at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:177)
 at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:157)
 at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager$1.run(LifecycleManager.java:207)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes, perhaps embedded inside the target plugin 'at.example.confluence.exampleplugin.example-plugin'; embedding scanner-annotations is not supported since scanner version 2.0. Use 'mvn dependency:tree' and ensure the atlassian-spring-scanner-annotation dependency in your plugin has <scope>provided</scope>, not 'runtime' or 'compile', and you have NO dependency on atlassian-spring-scanner-runtime.
 at com.atlassian.plugin.spring.scanner.runtime.impl.AtlassianScannerBeanDefinitionParser.checkScannerRuntimeIsNotEmbeddedInBundle(AtlassianScannerBeanDefinitionParser.java:198)
 at com.atlassian.plugin.spring.scanner.runtime.impl.AtlassianScannerBeanDefinitionParser.parse(AtlassianScannerBeanDefinitionParser.java:60)
 at org.springframework.beans.factory.xml.NamespaceHandlerSupport.parse(NamespaceHandlerSupport.java:74)
 at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1411)
 at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
 at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:168)
 at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
 at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
 at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
 at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
 ... 20 more

The Atlassian-Support-Team told us:

We had similar cases where we found the trigger was from a couple of conflicting third party plugins which have dependencies on the system plugin results in both third party and systems plugins failing to be enabled during the startup.

We have both, 3rd party and custom plugins installed in the System.

How can "unsatisfied dependencies: none" cause such behaviour?

Here is the pom.xml for one of them:

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>at.example.confluence.exampleplugin</groupId>
    <artifactId>example-plugin</artifactId>
    <version>${plugin.version}</version>
    <organization>
        <name>Example Company</name>
        <url>http://example.at</url>
    </organization>
    <name>Example Plugin</name>
    <description>Example Plugin</description>
    <packaging>atlassian-plugin</packaging>

    <properties>
        <confluence.version>6.7.2</confluence.version>
        <confluence.data.version>6.7.2</confluence.data.version>
        <amps.version>6.3.14</amps.version>
        <atlassian.spring.scanner.version>2.1.0</atlassian.spring.scanner.version>
        <atlassian.plugin.key>${project.groupId}.${project.artifactId}</atlassian.plugin.key>
        <base.version>1.4.4</base.version>
        <plugin.version>${base.version}</plugin.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <profiles>
        <profile>
            <id>special</id>
            <properties>
                <profile.name>special</profile.name>
                <plugin.version>${base.version}-special</plugin.version>
            </properties>
        </profile>
    </profiles>

    <dependencies>
        <dependency>
            <groupId>com.atlassian.confluence</groupId>
            <artifactId>confluence</artifactId>
            <version>${confluence.version}</version>
            <scope>provided</scope>
            <exclusions>
                <exclusion>
                    <groupId>com.atlassian.analytics</groupId>
                    <artifactId>analytics-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.mail</groupId>
                    <artifactId>mail</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>com.atlassian.plugin</groupId>
            <artifactId>atlassian-spring-scanner-annotation</artifactId>
            <version>${atlassian.spring.scanner.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>javax.inject</groupId>
            <artifactId>javax.inject</artifactId>
            <version>1</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>javax.ws.rs</groupId>
            <artifactId>jsr311-api</artifactId>
            <version>1.1.1</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.0.1</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.0</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.4</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>4.3.5.RELEASE</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>
    <build>
        <resources>
            <resource>
                <directory>src/main/profiles/${profile.name}</directory>
                <includes>
                    <include>*.properties</include>
                </includes>
            </resource>
            <resource>
                <directory>src/main/resources</directory>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <groupId>com.atlassian.plugin</groupId>
                <artifactId>atlassian-spring-scanner-maven-plugin</artifactId>
                <version>${atlassian.spring.scanner.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>atlassian-spring-scanner</goal>
                        </goals>
                        <phase>process-classes</phase>
                    </execution>
                </executions>
                <configuration>
                    <verbose>true</verbose>
                </configuration>
            </plugin>
            <plugin>
                <groupId>com.atlassian.maven.plugins</groupId>
                <artifactId>maven-confluence-plugin</artifactId>
                <version>${amps.version}</version>
                <extensions>true</extensions>
                <configuration>
                    <productVersion>${confluence.version}</productVersion>
                    <productDataVersion>${confluence.data.version}</productDataVersion>
                    <enableQuickReload>true</enableQuickReload>
                    <enableFastdev>false</enableFastdev>
                    <extractDependencies>false</extractDependencies>
                    <allowGoogleTracking>false</allowGoogleTracking>
                    <log4jProperties>src/log4j/log4j.properties</log4jProperties>
                    <!-- See here for an explanation of default instructions: -->
                    <!-- https://developer.atlassian.com/docs/advanced-topics/configuration-of-instructions-in-atlassian-plugins -->
                    <instructions>
                        <Atlassian-Plugin-Key>${atlassian.plugin.key}</Atlassian-Plugin-Key>
                        <!-- Add package to export here -->
                        <Export-Package>at.example.confluence.exampleplugin.api,</Export-Package>
                        <!-- Add package import here -->
                        <Import-Package>com.atlassian.confluence.*;resolution:="optional", *</Import-Package>
                        <!-- Ensure plugin is spring powered -->
                        <Spring-Context>*</Spring-Context>
                    </instructions>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-resources-plugin</artifactId>
                <version>3.0.2</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.6.1</version>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Is there anything wrong that could lead to the aforementioned exception?

Can anybody help us clarifying if this exception is even related to bugs in pom.xml?

Does anybody have similar experiences with custom plugins in confluence (eg. not loading correctly, BeanDefinitionStoreException, InvalidStateException)?

[Minh Tran]

@Matthias Steinböck Can i see your atlassian-plugin.xml?

[Matthias Steinböck]

Hey @Minh Tran thanks for your help!

here is the atlassian-plugin.xml. Do you think we did something wrong there?

<?xml version="1.0" encoding="UTF-8"?>

<atlassian-plugin key="${project.groupId}.${project.artifactId}" name="${project.name}" plugins-version="2">
    <plugin-info>
        <description key="at.example.confluence.exampleplugin.example-plugin.description" />
        <version>${project.version}</version>
        <vendor name="${project.organization.name}" url="${project.organization.url}" />
        <param name="plugin-icon">images/icon.png</param>
        <param name="plugin-logo">images/logo.png</param>
    </plugin-info>

    <!-- add our i18n resource -->
    <resource type="i18n" name="i18n" location="example-plugin"/>
    <resource type="download" name="images/" location="/images"/>

    <!-- add our web resources -->
    <web-resource key="example-plugin-resources" name="example-plugin Web Resources">
        <dependency>com.atlassian.auiplugin:ajs</dependency>
        <resource type="download" name="example-plugin.js" location="/js/example-plugin.js"/>
        <resource type="download" name="example-plugin.css" location="/css/example-plugin.css" />

        <context>example-plugin</context>
        <context>atl.general</context>
    </web-resource>

    <web-resource key="css-pdf-resources" name="example-plugin PDF Resources">
        <resource type="download" name="example-plugin.css" location="/css/example-plugin.css" />
        <context>example-plugin</context>
        <context>atl.general</context>
    </web-resource>

    <!-- macro that is placed on some pages -->
    <xhtml-macro name="example-plugin-macro" class="at.example.confluence.exampleplugin.macro.ExamplePluginMacro" key="example-plugin-macro"
                 icon="/download/resources/${project.groupId}.${project.artifactId}/images/macro.png"
            >
        <parameters>
            <parameter name="defaulttitle" type="string">
                <option key="showNameInPlaceholder" value="false" />
                <option key="showValueInPlaceholder" value="true" />
            </parameter>
            <parameter name="defaultcolor" type="string">
                <option key="showNameInPlaceholder" value="false" />
                <option key="showValueInPlaceholder" value="false" />
            </parameter>
            <parameter name="state" type="string" default="">
                <option key="showNameInPlaceholder" value="false" />
                <option key="showValueInPlaceholder" value="false" />
            </parameter>
            <parameter name="layout" type="enum" default="">
                <option key="showNameInPlaceholder" value="false" />
                <option key="showValueInPlaceholder" value="false" />
                <value name="subtle" />
                <value name="prominent" />
            </parameter>
        </parameters>
        <description key="at.example.confluence.exampleplugin.example-plugin.example-plugin-macro.desc" />
    </xhtml-macro>

    <!-- action that is called via ajax by the frontend-code the macro renders -->
    <xwork name="example-plugin-xwork" key="example-plugin-xwork">
        <description key="example-plugin-xwork.link.desc">Some action called by JavaScript.</description>
        <package name="example-plugin" extends="default" namespace="/plugins/example-plugin">
            <default-interceptor-ref name="defaultStack"/>
            <action name="do-action" class="at.example.confluence.exampleplugin.ExamplePluginAction">
                <result name="success" type="redirect">${page.urlPath}</result>
                <result name="json" type="json" />
            </action>
        </package>
    </xwork>

    <!-- administration servlet for manual actions -->
    <servlet key="manual-exampleplugin-update" i18n-name-key="at.example.confluence.exampleplugin.servlet" class="at.example.confluence.exampleplugin.servlet.UpdateServlet">
        <url-pattern>/updateExamplePluginMacros</url-pattern>
    </servlet>

</atlassian-plugin>

[Minh Tran]

Ohh you are using atlassian-spring-scanner 2.1.0. Have you upgraded it recently too?

[Matthias Steinböck]

Yes it is possible that the problem exists since the upgrade. But i have no evidence to support this. Did you have Problems with 2+? Did you switch back to an older one?

[Minh Tran]

@Matthias Steinböck You can follow the guideline here to upgrade your plugin to be compatible https://bitbucket.org/atlassian/atlassian-spring-scanner

[Matthias Steinböck]

@Minh Tran i followed the guide to the point.

I don't see any errors in my pom.xml or plugin-context.xml. Here it is for verification:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:atlassian-scanner="http://www.atlassian.com/schema/atlassian-scanner/2"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
       http://www.atlassian.com/schema/atlassian-scanner/2
       http://www.atlassian.com/schema/atlassian-scanner/2/atlassian-scanner.xsd">

    <!-- This file is loaded at plugin boot time, and loads all the components that don't have @Profile set.
         See also dynamic-spring-scanner.xml for loading of those that have @Profile("dynamic"). -->

    <atlassian-scanner:scan-indexes />

</beans>

So I already followed that guide. I also found the source code which is emitting this exception: https://bitbucket.org/atlassian/atlassian-spring-scanner/src/37f7d28a6e20e261dd878e83697e2411fbf93c74/atlassian-spring-scanner-runtime/src/main/java/com/atlassian/plugin/spring/scanner/runtime/impl/AtlassianScannerBeanDefinitionParser.java?at=atlassian-spring-scanner-parent-2.1.3&fileviewer=file-view-default#AtlassianScannerBeanDefinitionParser.java-180

But i have no clue why it is thrown.

[Sylvain Laurent]

Did you find a solution to this issue?

I have a bitbucket plugin which worked fine until recently but now I have been told that some 5.11 BB installations suffer from the same issue as you with my plugin. I don't understand why...

[Matthias Steinböck]

Hey @Sylvain Laurent do you use atlassian spring scanner 2 or 1? we think this is the problem. We downgraded and are currently testing it.

[Sylvain Laurent]

@Matthias Steinböck I'm using version 2.1.3.

I've check my plugin, it does not contain any spring scanner jars. I just have the import in the MANIFEST.MF which seems correct.

I really don't understand what's going on

[Sylvain Laurent]

@Minh Tran could the error come from this problem https://jira.atlassian.com/browse/CONFSERVER-55916 ? it seems a good candidate!