JIRA startup failed

[saif-mahmud]

Hi,

Jira failing to start after an upgrade from 7.13 to 8.0.1.

I have checked permissions, database connection.. all seems to be correct. Not seeing any info in the log file to trace the issue. Here's the log:

25-Feb-2019 04:17:01.101 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+PrintGCCause

25-Feb-2019 04:17:01.101 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=

25-Feb-2019 04:17:01.101 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/atlassian/jira

25-Feb-2019 04:17:01.101 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/atlassian/jira

25-Feb-2019 04:17:01.102 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/atlassian/jira/temp

25-Feb-2019 04:17:01.102 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]

25-Feb-2019 04:17:01.301 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]

25-Feb-2019 04:17:01.324 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 863 ms

25-Feb-2019 04:17:01.336 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]

25-Feb-2019 04:17:01.337 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.35

2019-02-25 04:17:05,915 localhost-startStop-1 INFO      [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/home/atlassian/jira-data' is validated and locked for exclusive use by this instance.

2019-02-25 04:17:05,984 JIRA-Bootstrap INFO      [c.a.jira.startup.JiraStartupLogger]

    ****************

    JIRA starting...

    ****************

2019-02-25 04:17:06,058 JIRA-Bootstrap INFO      [c.a.jira.startup.JiraStartupLogger]

___ Environment _____________________________

         JIRA Build                                    : 8.0.1#800009-sha1:f5186e157f3e26c0180673ce49d51cb56d8f7579

         Build Date                                    : Wed Feb 20 00:00:00 EST 2019

         JIRA Installation Type                        : Standalone

         Application Server                            : Apache Tomcat/8.5.35 - Servlet API 3.1

         Java Version                                  : 1.8.0_181 - Oracle Corporation

         Current Working Directory                     : /home/atlassian/jira/bin

         Maximum Allowable Memory                      : 1367MB

         Total Memory                                  : 797MB

         Free Memory                                   : 740MB

         Used Memory                                   : 57MB

         Memory Pool: Code Cache                       : Code Cache: init = 33554432(32768K) used = 10375936(10132K) committed = 33554432(32768K) max = 536870912(524288K)

         Memory Pool: Metaspace                        : Metaspace: init = 0(0K) used = 22451192(21924K) committed = 23199744(22656K) max = -1(-1K)

         Memory Pool: Compressed Class Space           : Compressed Class Space: init = 0(0K) used = 2588272(2527K) committed = 2752512(2688K) max = 1073741824(1048576K)

         Memory Pool: PS Eden Space                    : PS Eden Space: init = 134742016(131584K) used = 37884736(36996K) committed = 469237760(458240K) max = 493355008(481792K)

         Memory Pool: PS Survivor Space                : PS Survivor Space: init = 22020096(21504K) used = 0(0K) committed = 8912896(8704K) max = 8912896(8704K)

         Memory Pool: PS Old Gen                       : PS Old Gen: init = 358088704(349696K) used = 21470248(20967K) committed = 358088704(349696K) max = 1075314688(1050112K)

         JVM Input Arguments                           : -Djava.util.logging.config.file=/home/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx1538m -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -Datlassian.plugins.enable.wait=300 -XX:-OmitStackTraceInFastThrow -Djava.locale.providers=COMPAT -Datlassian.plugins.startup.options= -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xloggc:/home/atlassian/jira/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -Dignore.endorsed.dirs= -Dcatalina.base=/home/atlassian/jira -Dcatalina.home=/home/atlassian/jira -Djava.io.tmpdir=/home/atlassian/jira/temp

         Java Compatibility Information                : JIRA version = 8.0.1, Java Version = 1.8.0_181

2019-02-25 04:17:06,267 JIRA-Bootstrap INFO      [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /home/atlassian/jira-data/dbconfig.xml

2019-02-25 04:17:06,768 JIRA-Bootstrap INFO      [c.a.jira.startup.JiraStartupLogger] Running JIRA startup checks.

2019-02-25 04:17:06,769 JIRA-Bootstrap FATAL      [c.a.jira.startup.JiraStartupLogger] Startup check failed. JIRA will be locked.

2019-02-25 04:17:06,829 JIRA-Bootstrap INFO      [c.a.jira.startup.LauncherContextListener] Memory Usage:

    ---------------------------------------------------------------------------------

      Heap memory     :  Used:  182 MiB.  Committed:  798 MiB.  Max: 1505 MiB

      Non-heap memory :  Used:   41 MiB.  Committed:   64 MiB.  Max: 1536 MiB

    ---------------------------------------------------------------------------------

      TOTAL           :  Used:  223 MiB.  Committed:  861 MiB.  Max: 3041 MiB

    ---------------------------------------------------------------------------------

25-Feb-2019 04:17:18.600 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]

25-Feb-2019 04:17:18.611 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read

25-Feb-2019 04:17:18.636 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 17311 ms

[Mihai Schwarz]

Hi @saif-mahmud - have a look at the catalina and localhost logs, you will find the reason why your instance is getting locked. Location of the logs are in ...\Program Files\Atlassian\JIRA\logs if you are using Windows, if you are using Linux is  /var/atlassian/JIRA/logs.

[saif-mahmud]

Hi @Mihai Schwarz

localhost log file is empty. Nothing there.

catalina log has this:

25-Feb-2019 04:22:46.178 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [webwork.action.ActionContext$1] (value [webwork.action.ActionContext$1@76ad485]) and a value of type [webwork.action.ActionContext] (value [webwork.action.ActionContext@2bbb242a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

25-Feb-2019 04:22:46.179 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [webwork.action.ActionContext$1] (value [webwork.action.ActionContext$1@76ad485]) and a value of type [webwork.action.ActionContext] (value [webwork.action.ActionContext@3e8aab7e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

25-Feb-2019 04:22:46.179 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [webwork.action.ActionContext$1] (value [webwork.action.ActionContext$1@76ad485]) and a value of type [webwork.action.ActionContext] (value [webwork.action.ActionContext@56816c83]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

25-Feb-2019 04:22:46.192 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]

25-Feb-2019 04:22:46.200 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"

[Mihai Schwarz]

Hi @saif-mahmud - looks like you have some memory leaks.

These articles should help you fix the issue: https://confluence.atlassian.com/jirakb/jira-applications-crash-due-to-outofmemoryerror-java-heap-space-191069.html?_ga=2.220845208.189573661.1535439702-1125479650.1525245948

https://confluence.atlassian.com/jirakb/crashes-and-performance-issues-troubleshooting-203394749.html?_ga=2.220845208.189573661.1535439702-1125479650.1525245948

[saif-mahmud]

Hi @Mihai Schwarz

I have doubled Min and Max memory limit. Also, GC seems to be active and no error in GC logs. Jira still failing to pass start up check.

[Mihai Schwarz]

@saif-mahmudalso try this:

• Stop the JIRA instance.
• Delete JIRA_HOME/plugins/.bundled-plugins and JIRA_HOME/plugins/.osgi-plugins (just these two)
• Restart JIRA

[saif-mahmud]

Hi @Mihai Schwarz

Thanks. But that didn't solve the issue either.

I had to revert back to previous version and did a clean upgrade again. It worked this time. Everything is working now. Not sure what happened during the last upgrade.

[Alexis Robert]

Hi @saif-mahmud , 

the only error I see here is " Startup check failed. ", there is nothing else in the logs. 

Can you give a bit more details on how you did the upgrade ? What steps did you follow ? 

Thanks, 

--Alexis

[saif-mahmud]

Hi @Alexis Robert ,

Thanks for looking into this.

I have dowloaded the latest version of Jira core and selected upgrade an existing installation option. It went well without any error. I have some proxy and https settings in server.xml config file, I transferred those over to new server.xml and tried to start jira instance. But it kept failing.