Jira Web GUI stop working

[Weiqi Yao]

Hi Everyone,

Our Jira Web interface stop working since This Monday(4/17) we came back to work. I am new to Jira and have no idea, I tried to check the logs but barely find anything.

Can someone suggest some key word I can search in  atlassian-jira.log?

Or anyone knows how to troubleshoot?

Thanks!

[Weiqi Yao · Accepted solution]

I opened a tech support ticket with ATLASSIAN, and the only thing they can tell from the logs are just hang of jira. But I investigate and resolved myself. It was because of the jira/nginx integration issue. I re-do the process based on the official doc, and the website came back.

btw, The tech support team suggest to enable the core dump log on jira, I will look into that maybe helpful to further troubleshoot.

Thanks.

[Nic Brough -Adaptavist-]

I am afraid the answer is that you are probably going to have to read the logs.

There's no way of knowing what the cause of the failure is, but the logs will tell you.  I can't tell you what to search for in a log until I know what the cause is, and I can't know what the cause is until you've read the logs.

I'd start with a description of the error here ("gui stop working" is not enough detail.  Assumign the dashboard, is it blank?  Error message?  Corrupt?  Looks fine but clicking does nothing?) and then a look at the most recent errors in the log

[Weiqi Yao]

Sorry for the unclear description, the Web GUI just simply stop respond. And I checked the log file and the log file become pretty tricky since here

2018-04-16 01:36:46,117 Caesium-1-2 INFO ServiceRunner     [c.a.crowd.directory.DbCachingRemoteDirectory] INCREMENTAL synchronisation complete for directory [ 10100 ] in [ 180ms ]
2018-04-16 02:03:08,755 localhost-startStop-2 INFO      [c.a.jira.startup.DefaultJiraLauncher] Stopping launchers
2018-04-16 02:03:08,818 localhost-startStop-2 INFO      [c.a.plugin.manager.DefaultPluginManager] Preparing to shut down the plugin system
2018-04-16 02:03:08,856 localhost-startStop-2 INFO      [c.a.p.web.springmvc.DispatcherServlet$1] Closing Root WebApplicationContext: startup date [Wed Oct 25 14:22:58 UTC 2017]; parent: NonValidatingOsgiBundleXmlApplicationContext(bundle=com.atlassian.crowd.embedded-crowd-admin-plugin, config=osgibundle:/META-INF/spring/*.xml)
2018-04-16 02:03:08,881 localhost-startStop-2 INFO      [c.a.plugin.manager.DefaultPluginManager] Shutting down the plugin system

 After this, the Jira just start by itself. I believe after thisthe Jira WebGUI stopped.

Thanks

[Nic Brough -Adaptavist-]

Sorry, but "Web Gui stopped" is not an answer.  What stopped, what does stopped mean? 

You'll need to read more of the logs - that's just part of it shutting down.

[Weiqi Yao]

Hi Nic, appreciate fo your reply, I check the log and really didn't found anything attracts. I also try to access the web UI using 8081 instead of 8080, and it worked, but I couldnot login. Is this could be something wrong with my integration process with Nginx?

Here is my server.xml config, hope this helps:

<?xml version="1.0" encoding="utf-8"?>
<!--
   ====================================================================================

   Atlassian JIRA Standalone Edition Tomcat Configuration.


   See the following for more information

   http://confluence.atlassian.com/display/JIRA/Configuring+JIRA+Standalone

   ====================================================================================
 -->
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<Server port="8005" shutdown="SHUTDOWN">
    <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
    <!-- Security listener. Documentation at /docs/config/listeners.html
    <Listener className="org.apache.catalina.security.SecurityListener" />
    -->
    <!--APR library loader. Documentation at /docs/apr.html -->
    <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
    <!-- Prevent memory leaks due to use of particular java/javax APIs-->
    <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
    <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
    <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

    <!-- Global JNDI resources
         Documentation at /docs/jndi-resources-howto.html
    -->

    <!-- A "Service" is a collection of one or more "Connectors" that share
        a single "Container" Note:  A "Service" is not itself a "Container",
        so you may not define subcomponents such as "Valves" at this level.
        Documentation at /docs/config/service.html
    -->
    <Service name="Catalina">

        <Connector port="8080"
          maxThreads="150"
          minSpareThreads="25"
          connectionTimeout="20000"
          enableLookups="false"
          maxHttpHeaderSize="8192"
          protocol="HTTP/1.1"
          useBodyEncodingForURI="true"
          redirectPort="8443"
          acceptCount="100"
          disableUploadTimeout="true"/>

<!-- OPTIONAL,Nginx Proxy Connector with https -->
        <Connector port="8081"
          maxThreads="150"
          minSpareThreads="25"
          connectionTimeout="20000"
          enableLookups="false"
          maxHttpHeaderSize="8192"
          protocol="HTTP/1.1"
          useBodyEncodingForURI="true"
          redirectPort="8443"
          acceptCount="100"
          disableUploadTimeout="true"
          proxyName="jira.intelligentintercept.com"
          proxyPort="443"
          scheme="https"
          secure="true"/>



        <!--
        ====================================================================================

        For full steps on running JIRA over SSL or HTTPS for production and testing, see:
            http://confluence.atlassian.com/display/JIRA/Running+JIRA+over+SSL+or+HTTPS
        and
            http://tomcat.apache.org/tomcat-8.0-doc/ssl-howto.html

        A quicker method can be found below, which we recommend only for evaluation and demonstration:

            * Uncomment the Connector below
            * Execute:
                %JAVA_HOME%\bin\keytool -genkey -alias tomcat -keyalg RSA (Windows)�
                JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA (Unix)�
                with a password value of "changeit" for both the certificate and the keystore itself.
            * If you are on JDK1.3 or earlier, download and install JSSE 1.0.2 or later, and put the JAR files into "$JAVA_HOME/jre/lib/ext"
            * Restart and visit https://localhost:8443/

        ====================================================================================
        -->
        <!--
            <Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol"
              maxHttpHeaderSize="8192" SSLEnabled="true"
              maxThreads="150" minSpareThreads="25"
              enableLookups="false" disableUploadTimeout="true"
              acceptCount="100" scheme="https" secure="true"
              clientAuth="false" sslProtocol="TLS" useBodyEncodingForURI="true"/>
        -->


        <!--
         ====================================================================================

         If you have Apache AJP Connector (mod_ajp) as a proxy in front of JIRA you should uncomment the following connector configuration line

         See the following for more information :

            http://confluence.atlassian.com/display/JIRA/Configuring+Apache+Reverse+Proxy+Using+the+AJP+Protocol

         ====================================================================================
        -->

        <!--
              <Connector port="8009" redirectPort="8443" enableLookups="false" protocol="AJP/1.3" URIEncoding="UTF-8"/>
        -->

        <Engine name="Catalina" defaultHost="localhost">
            <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">

                <Context path="" docBase="${catalina.home}/atlassian-jira" reloadable="false" useHttpOnly="true">


                    <!--
                     ====================================================================================

                     Note, you no longer configure your database driver or connection parameters here.
                     These are configured through the UI during application setup.

                     ====================================================================================
                    -->

                    <Resource name="UserTransaction" auth="Container" type="javax.transaction.UserTransaction"
                              factory="org.objectweb.jotm.UserTransactionFactory" jotm.timeout="60"/>
                    <Manager pathname=""/>
                </Context>

            </Host>

            <!--
                ====================================================================================

                 Access Logging.

                 This should produce access_log.<date> files in the 'logs' directory.

                 The output access log lies has the following fields :

                 IP Request_Id User Timestamp  "HTTP_Method URL Protocol_Version" HTTP_Status_Code ResponseSize_in_Bytes RequestTime_In_Millis Referer User_Agent ASESSIONID

                 eg :

                 192.168.3.238 1243466536012x12x1 admin [28/May/2009:09:22:17 +1000] "GET /jira/secure/admin/jira/IndexProgress.jspa?taskId=1 HTTP/1.1" 200 24267 1070 "http://carltondraught.sydney.atlassian.com:8090/jira/secure/admin/jira/IndexAdmin.jspa" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10" "C2C99B632EE0F41E90F8EF7A201F6A78"

                 NOTES:

                 The RequestId is a millis_since_epoch plus request number plus number of concurrent users

                 The Request time is in milliseconds

                 The ASESSIONID is an hash of the JSESSIONID and hence is safe to publish within logs.  A session cannot be reconstructed from it.

                 See http://tomcat.apache.org/tomcat-6.0-doc/config/valve.html for more information on Tomcat Access Log Valves

                ====================================================================================

            -->
            <Valve className="org.apache.catalina.valves.AccessLogValve"
                   pattern="%a %{jira.request.id}r %{jira.request.username}r %t &quot;%m %U%q %H&quot; %s %b %D &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot; &quot;%{jira.request.assession.id}r&quot;"/>

        </Engine>
    </Service>
</Server>

[Nic Brough -Adaptavist-]

If your Jira is not responding to you, but is running, then you need to read your logs.

I know they're long and dry and dull, and not easy to read or spot where something is going wrong, but there is no other way to know what is going wrong.

Please, go back to them and read them.  There are errors in them that tell you what is failing.

[Weiqi Yao]

Hi Nic,

Thanks for your suggestion, and I did go back and read the log file again. Found two suspected pats.

2018-04-16 01:36:46,117 Caesium-1-2 INFO ServiceRunner     [c.a.crowd.directory.DbCachingRemoteDirectory] INCREMENTAL synchronisation complete for directory [ 10100 ] in [ 180ms ]
2018-04-16 02:03:08,755 localhost-startStop-2 INFO      [c.a.jira.startup.DefaultJiraLauncher] Stopping launchers
2018-04-16 02:03:08,818 localhost-startStop-2 INFO      [c.a.plugin.manager.DefaultPluginManager] Preparing to shut down the plugin system
2018-04-16 02:03:08,856 localhost-startStop-2 INFO      [c.a.p.web.springmvc.DispatcherServlet$1] Closing Root WebApplicationContext: startup date [Wed Oct 25 14:22:58 UTC 2017]; parent: NonValidatingOsgiBundleXmlApplicationContext(bundle=com.atlassian.crowd.embedded-crowd-admin-plugin, config=osgibundle:/META-INF/spring/*.xml)
2018-04-16 02:03:08,881 localhost-startStop-2 INFO      [c.a.plugin.manager.DefaultPluginManager] Shutting down the plugin system

Above shows the Jira has no activity for about half hour from 1:36 to 2:03 and then stop the launchers automatically(due to the time period, it can not be anyone logged in and shut down it manully.)

2018-04-16 02:06:29,237 HealthCheck:thread-4 ERROR ServiceRunner     [c.a.j.p.healthcheck.support.BaseUrlHealthCheck] An error occurred when performing the Base URL healthcheck: 
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)

I found an ERROR happend right after the Jira finish launching:

PKIX path building failed

I am not sure what it is, is this related to the SSL certificate, we've tried to add https function to Jira, but failed. And from then we never use https again, and stay http.

Any suggestions on this?

Thank you and appreicate!

[Nic Brough -Adaptavist-]

The first log is telling you that someone or something told it to shut down at 2:03.  Have a look at the application log for that time.

The second log is not a crash report, it's telling you that there's something wrong with your https setup.  If you're not using it now, you can ignore it (although I'd still recommend fixing the root problem)

[Weiqi Yao]

Thanks Nic,

I checked the catalina.out log and found:

16-Apr-2018 02:03:08.665 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
16-Apr-2018 02:03:08.666 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
16-Apr-2018 02:03:08.666 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
2018-04-16 02:03:08,755 localhost-startStop-2 INFO      [c.a.jira.startup.DefaultJiraLauncher] Stopping launchers
2018-04-16 02:03:08,818 localhost-startStop-2 INFO      [c.a.plugin.manager.DefaultPluginManager] Preparing to shut down the plugin system

I also use wireshark to capture traffic between me and the jira server, looks like the the server close the connection right after the 3-way handshake.

[Alexis Robert]

Hello, 

did you try to just stop and restart Jira on the server ? Usually the logs will have some meaningful information, you can check them at <JIRA_HOME>/logs/atlassian-confluence.log

Let me know if it works.

Cheers, 

--Alexis

[Weiqi Yao]

Hi Alexis,

I tried stop/start jira locally on the server and still not working after that. And I did check the atlassian-confluence.log file, but I don't find anything intersting in that, is there a way I and upload the log file here?