Getting out of memory issues

[Srikanth Gogineni]

Hi Team,

we are often getting out of memory issues and which causing the application down. could any one suggest regarding increase/decreas in JVM memory. below are our application specifiatons

Jira Version : 6.2.7

system Memory : 32 GB

JVM Memory : 8GB

database is SQL Server 

we have around 400000 issues with 8 projects .

please let me know if any other details required.

Thanks in Advance.

[Nic Brough -Adaptavist-]

What exactly are the memory errors you get, and is there a trigger?  For example, it happens when you run an export or a particular board?

Don't just try throwing memory or resource at this, you can make it worse if you don't know  why you're doing it and how it might improve.

[Srikanth Gogineni]

Hi Nic/ Lennart,

we are getting the below memory errors, when the users performing some actions like searching,editing, moving the issues.

/secure/ClonePlusDetails.jspa [jira.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Wait attempt timed out - waited 30000 milliseconds

Uncaught exception thrown by REST service: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded

rest/issueNav/1/issueTable [NoModule] There was an error getting a DBCP datasource.
java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool

and the CPU usage is high(around 70 to 90%) most of the time and the application is getting down during 6 pm to 7 pm IST time .

we have McAfee Service installed in the Server but that service is not taking much allocation from CPU 

Let me know if you need any other details .

[Lennart]

• exlude JIRA files and folders from scanning (again, to be in support): https://confluence.atlassian.com/jirakb/anti-virus-in-jira-applications-398393384.html
  • your antivirus could be the cause for your problems - check this documentation too
• a higher cpu usage of JAVA could also be related to gc running too frequently (@Nic Brough -Adaptavist- is totally right, a bigger HeapSpace doesn't always help and is sometimes even worse - still, your issues could also be related to HeapSpace sizing)
• check this documentation on how to stabilize your db-connection
• again, update your JIRA instance to JIRA 7.6 (this should give you some performance and bring you back into developer-support)

My guess: Your antivirus causes the main issues.

[Srikanth Gogineni]

Hi Lennart/ Nic,

below is the JVM configuration, please suggest me to add/remove anything from this configuration.

-Dcatalina.base=C:\Program Files\Atlassian\JIRA
-Dcatalina.home=C:\Program Files\Atlassian\JIRA
-Djava.endorsed.dirs=C:\Program Files\Atlassian\JIRA\endorsed
-Djava.awt.headless=true
-Datlassian.standalone=JIRA
-Dmail.mime.decodeparameters=true
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
-Djava.io.tmpdir=C:\Program Files\Atlassian\JIRA\temp
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=C:\Program Files\Atlassian\JIRA\conf\logging.properties
-XX:MaxPermSize=1024m
-Dhibernate.show_sql=false
-Dfile.encoding=UTF8
-XX:+UseParallelGC
-Duser.language=en
-Duser.country=US

here is the DBconfig configuration

pool-min-size>80</pool-min-size>
<pool-max-size>80</pool-max-size>
<pool-max-wait>60000</pool-max-wait>
<validation-query>select 1</validation-query>
<pool-max-idle>80</pool-max-idle>
<pool-remove-abandoned>true</pool-remove-abandoned>
<pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout>
</jdbc-datasource>

we are facing severe crashes which impacting business very badly

[Nic Brough -Adaptavist-]

Debugging and tuning JVMs is not a small or simple subject.

You've only given us part of the JVM settings, and we'd need a lot more from the environment, we don't know the java version, if you're using a proxy, your logging levels (which usually need to be increased to help track down crashes), the exact symptoms (sudden crashes, creeping slowness for example), the profile of usage around crashes, any pattern to the crashing (like every tuesday at 4am), and so-on.

Dan did a talk at my first Summit on this - https://www.atlassian.com/company/events/summit-us/watch-sessions/2013/archives/inside-the-massive-team/the-not-so-dark-art-if-atlassian-performance-tuning .  This is 5 years old, but the principles are still right.

If you're experiencing regular crashes though, I would look for expert help as well - there are plenty of people who can help you with JVMs, but I would recommend sticking with experts in the Atlassian stack.

[Srikanth Gogineni]

Hi Nic/Lennart,

Finally I have overcome this outage error by increasing the memory to 16GB and connection pool from 80 to 100. 

Now the application is working fine as expected. Thank you both for the Support.

Regards

Srikanth G

[Lennart]

@Srikanth Gogineni:

• update JIRA to a supported version (i.e. 7.6 Enterprise) <- !
• create gc-logfiles and investigate on them
• raise your JVM heapspace to i.e. 16GB (same min / max)