Jira can't start due to atlassian-authentication-plugin timeout

[Hakujou]

Hello,

I run Jira using official Jira Docker container: https://hub.docker.com/r/atlassian/jira-software

Since yesterday, Jira refuses to start without error, only giving up due to plugin timeout:

2020-03-23 13:41:50,668+0000 localhost-startStop-1 INFO [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/var/atlassian/application-data/jira' is validated and locked for exclusive use by this instance.
2020-03-23 13:41:50,782+0000 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

****************
Jira starting...
****************

2020-03-23 13:41:50,902+0000 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

___ Environment _____________________________

JIRA Build : 8.8.0#808000-sha1:e2c7e59ae165efc6ad6b529150e24d091b9947bf
Build Date : Thu Mar 19 00:00:00 GMT 2020
JIRA Installation Type : Standalone
Application Server : Apache Tomcat/8.5.42 - Servlet API 3.1
Java Version : 1.8.0_242 - AdoptOpenJDK
Current Working Directory : /var/atlassian/application-data/jira
Maximum Allowable Memory : 3641MB
Total Memory : 654MB
Free Memory : 449MB
Used Memory : 205MB
Memory Pool: Code Cache : Code Cache: init = 33554432(32768K) used = 11242432(10978K) committed = 33554432(32768K) max = 536870912(524288K)
Memory Pool: Metaspace : Metaspace: init = 0(0K) used = 20867384(20378K) committed = 21364736(20864K) max = -1(-1K)
Memory Pool: Compressed Class Space : Compressed Class Space: init = 0(0K) used = 2335904(2281K) committed = 2490368(2432K) max = 1073741824(1048576K)
Memory Pool: PS Eden Space : PS Eden Space: init = 100663296(98304K) used = 186208840(181844K) committed = 400556032(391168K) max = 1396178944(1363456K)
Memory Pool: PS Survivor Space : PS Survivor Space: init = 16777216(16384K) used = 16166896(15787K) committed = 17301504(16896K) max = 17301504(16896K)
Memory Pool: PS Old Gen : PS Old Gen: init = 268435456(262144K) used = 12261048(11973K) committed = 268435456(262144K) max = 2863661056(2796544K)
JVM Input Arguments : -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms384m -Xmx4096m -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -Djavax.net.ssl.trustStore=/var/atlassian/application-data/jira/cacerts -XX:-OmitStackTraceInFastThrow -Djava.locale.providers=COMPAT -Djira.home=/var/atlassian/application-data/jira -Datlassian.plugins.startup.options=-fg -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xloggc:/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -XX:+ExplicitGCInvokesConcurrent -Dignore.endorsed.dirs= -Dcatalina.base=/opt/atlassian/jira -Dcatalina.home=/opt/atlassian/jira -Djava.io.tmpdir=/opt/atlassian/jira/temp
Java Compatibility Information : JIRA version = 8.8.0, Java Version = 1.8.0_242

...

2020-03-23 13:42:04,469+0000 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigurationManagerImpl] The database is configured. Now running Database Checklist Launcher
2020-03-23 13:42:04,476+0000 JIRA-Bootstrap INFO [c.a.jira.startup.DatabaseChecklistLauncher] Jira database startup checks completed successfully.
2020-03-23 13:42:04,478+0000 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigurationManagerImpl] The database is configured. Now running Post database-configuration launchers
2020-03-23 13:42:07,114+0000 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira/dbconfig.xml
2020-03-23 13:42:07,117+0000 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigHandler] Trying to get encrypted password from xml and decrypt it
2020-03-23 13:42:07,118+0000 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigHandler] Database password decryption not performed.
2020-03-23 13:42:09,219+0000 JIRA-Bootstrap INFO [c.a.jira.i18n.CachingI18nFactory] [i18n-caching-factory] Created
2020-03-23 13:42:10,320+0000 JIRA-Bootstrap INFO [c.a.j.c.embedded.ofbiz.OfBizUserDao] Using full user cache
2020-03-23 13:42:13,542+0000 JIRA-Bootstrap INFO [c.a.jira.upgrade.PluginUpgradeListener] [plugin-upgrade-listener] Created
2020-03-23 13:42:13,543+0000 JIRA-Bootstrap INFO [c.a.j.c.cache.pauser.NonClusteredReplicationPauserManager] Non-clustered mode: ReplicationPauserManager implemented by NonClusteredReplicationPauserManager
2020-03-23 13:42:13,545+0000 JIRA-Bootstrap INFO [c.a.jira.plugin.JiraCacheResetter] [jira-cache-reseter] Created and registered for events
2020-03-23 13:42:13,551+0000 JIRA-Bootstrap INFO [c.a.jira.plugin.PluginTransactionListener] [plugin-transaction] Created, registered for events and schedulled stats job
2020-03-23 13:42:13,583+0000 JIRA-Bootstrap INFO [c.a.jira.i18n.CachingI18nFactory] [i18n-caching-factory] Registering for events
2020-03-23 13:42:13,609+0000 JIRA-Bootstrap INFO [c.a.jira.plugin.JiraCacheResetter] [jira-cache-reseter] Initialized.
2020-03-23 13:42:13,610+0000 JIRA-Bootstrap INFO [c.a.jira.plugin.PluginTransactionListener] [plugin-transaction] Initialized.
2020-03-23 13:42:13,720+0000 JIRA-Bootstrap INFO [c.a.jira.startup.DatabaseLauncher] Database transactions enabled: true
2020-03-23 13:42:13,721+0000 JIRA-Bootstrap INFO [c.a.jira.startup.DatabaseLauncher] Using JIRA's default for database transaction isolation level: 2
2020-03-23 13:42:13,731+0000 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

Database configuration OK

...

___ Starting the JIRA Plugin System _________________
020-03-23 13:42:53,931+0000 JIRA-Bootstrap INFO [c.a.plugin.util.WaitUntil] Plugins that have yet to be enabled: (1): [com.atlassian.plugins.authentication.atlassian-authentication-plugin], 

...

2020-03-23 13:47:38,051+0000 JIRA-Bootstrap INFO [c.a.plugin.util.WaitUntil] Plugins that have yet to be enabled: (1): [com.atlassian.plugins.authentication.atlassian-authentication-plugin], 1 seconds remaining
2020-03-23 13:47:39,051+0000 JIRA-Bootstrap INFO [c.a.plugin.util.WaitUntil] Plugins that have yet to be enabled: (1): [com.atlassian.plugins.authentication.atlassian-authentication-plugin], 0 seconds remaining
2020-03-23 13:47:40,130+0000 JIRA-Bootstrap ERROR [c.a.plugin.manager.PluginEnabler] Unable to start the following plugins due to timeout while waiting for plugin to enable: com.atlassian.plugins.authentication.atlassian-authentication-plugin

I tried to rollback the Docker container to older versions (down to 8.4.0) without any changes. 

JVM_MAXIMUM_MEMORY is set to 4096m, which should be plenty (considering there's less than 50 tickets on the platform).

Are there any clues why this particular plugin can't boot ? I can't figure out why. Please let me know if you need additional details.

Thanks a lot.

[Mike Rathwell]

Hi @Hakujou ,

Is that the only plugin that is not loading before Jira gives up trying to load? I might suggest doing the manual plugin cache clear documented on this page. 

Since this broke SO hard, are you sure all the volumes necessary for Jira to start are being mounted by the container?

Finally, assuming you are not just using the testing H2 database and pointing at a remote database of some sort, rolling back versions in Jira is almost always bad juju. Very often the schema of the database or even just how the content in the current schema is stored can be altered with an upgrade. Trying to run an older version of Jira against a dataset from a newer version seldom ends well.

[Hakujou]

Hi,

Sorry I should have mentioned it, yes this is the only plugin that won't load. I already cleared the cache (well, except inside the "Work" folder since I don't have one), without changes.

For the database, indeed I'm using Postgres (also inside a docker container), I was just debugging (I performed a database backup before to avoid corruption due to unproper rollback, as you mentioned).

Thanks for your help !