JIRA becomes unresponsive in the middle of the day

[Deleted user]

Our small company with about 15,000 issues is, out of the blue, experiencing seemingly random times where the application becomes unresponsive (not completely, but minute+ page loads that normally take a second) with high cpu usage and no indications that I am aware of to see how long it will take (today it was about 4pm). When this isn't happening, the service runs reliably and responsively enough.

I'm able to stop and restart JIRA to make it available for our people to use again but that isn't a good thing to have to do every couple days.

Any thoughts on where to begin isolating the cause for this? Google searches turn up conflicting, overly technically-deep advice.

JIRA version 7.13.0

Thanks!

[Robert Dzido _Almarise_ · Accepted solution]

Hi Nick,

Symptoms such as high CPU usage and no response usually are connected to memory usage by JVM, but please provide some more details on your JVM parameters (available in System->System Info section) and overall system configuration (operating system, memory, cpu, etc).

[Deleted user]

Hello Robert,

This is running on a m5a.large Amazon instance (2 vpu, 8GB RAM).

This is what I have from the System Info Section:

• System Date Wednesday, 22 Jan 2020
• System Time 18:01:23 -0500
• Current Working Directory /opt/atlassian/jira
• Java Version 1.8.0_181
• Java Vendor Oracle Corporation
• JVM Version 1.8
• JVM Vendor Oracle Corporation
• JVM Implementation Version 25.181-b13
• Java Runtime Java(TM) SE Runtime Environment
• Java VM Java HotSpot(TM) 64-Bit Server VM
• User Name jira
• User Timezone America/New_York
• User Locale English (United States)
• System Encoding UTF-8
• Operating System Linux 4.14.158-101.185.amzn1.x86_64
• OS Architecture amd64
• Application Server Container Apache Tomcat/8.5.32
• Database type mysql
• Database version 5.7.26-log
• Database JNDI address mysql jdbc:mysql://REDACTED/Jira?useUnicode=true&characterEncoding=UTF8
• Database URL jdbc:mysql://REDACTED/Jira?useUnicode=true&characterEncoding=UTF8
• Database driver MySQL Connector Java mysql-connector-java-5.1.39 ( Revision: 3289a357af6d09ecc1a10fd3c26e95183e5790ad )
• Database collation utf8_bin
• External user management OFF
• JVM Input Arguments -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms384m -Xmx1024m -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -Datlassian.plugins.enable.wait=300 -XX:-OmitStackTraceInFastThrow -Datlassian.plugins.startup.options= -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xloggc:/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -Dignore.endorsed.dirs= -Dcatalina.base=/opt/atlassian/jira -Dcatalina.home=/opt/atlassian/jira -Djava.io.tmpdir=/opt/atlassian/jira/temp
• Modified Files [Installation Type: Standalone] jira-application.properties, WEB-INF/urlrewrite.xml
• Removed Files [Installation Type: Standalone] There have been no removed files
• Clustering OFF
• Java VM Memory Statistics
  • Total Memory 946 MB
  • Free Memory 266 MB
  • Used Memory 680 MB
• Memory Graph
  • 28% Free (Total: 946 MB) (Force garbage collection)

Thanks!

[Robert Dzido _Almarise_]

You can try adding more heap memory to java.

Go to /opt/atlassian/jira/bin/setenv.sh

find line

JVM_MAXIMUM_MEMORY="1024m"

and replace with

JVM_MAXIMUM_MEMORY="2g"

Then restart jira service.

Do you have EazyBI or another similar add-on installed?

[Deleted user]

No idea what EazyBI is. Not using it.

We are using the following add-ons:

• Automation Lite for Jira
• Canned Responses for JIRA
• Checklist
• Jira Misc Workflow Extensions

I've increase the max memory to your recommended setting of "2g".

JIRA started in about half the time. Seems like a deficiency in the product that it doesn't do this automatically or have a non-editing-buried-config-files way of doing this.

Anyway, I guess we'll see.

Thanks!  ; )

[Deleted user]

So far so good...we'll give it another day and then I'll accept this answer if no further issues

[Deleted user]

Accepted as answer. Haven't seen the issue pop up again since implementation.

This answer also seemed to help with an upgrade attempt that failed the previous week due to add-ons not updating as expected.

[Aron Gombas _Midori_]

Isn't this possible that re-indexing is done in the background? (That puts rather resource-consuming load to any Jira, temporarily.)

[Deleted user]

How do I check that?

[Aron Gombas _Midori_]

Generally speaking, I'd go and check the Jira logs to see what's happening in the period when you experience the performance degradation.

If it is indexing, it will be there, if it's something else, it will also be there.

[Deleted user]

Hi Aron,

Thank you.

One of the issues with this software is the sheer amount of logging it collects, which I guess isn't in itself bad except one won't know where to start.

Just eyeballing:

• there are about 10 different log file types in /opt/atlassian/jira/logs
• about 16 different log file types in /var/atlassian/application-data/jira/log

Now, some research just now is pointing to /var/atlassian/application-data/jira/log/atlassian-jira.log as a primary log to use, however it's very difficult to read through (and seems to contain fairly low-level java debugging messages).

Any suggestions on something I can run on this headless server to help me parse through these files?

[Robert Dzido _Almarise_]

Hi @[deleted]

What you can find as a non-developer or system admin, you can go to Jira services (System->Services) and check out backup service. In most cases it's executed twice a day, at 1AM and 1PM and it can cause some performance bottlenecks, but after you increased heap memory it shall be fine.

You can modify Cron expression to run backup only at night by entering following expression: 0 0 5 1/1 * ? *

It will start backup at 5AM each day.

[Deleted user]

Hello Robert,

It says "Each Monday at 1:00 am" which would rule it out as a cause for this issue.

Doing a quick little research indicates this it is discouraged for production systems...and since we grab routine snapshots of the whole instance and the database on a fairly granular basis, I have decided to remove this service.