Jira configured to use 6GB of RAM in an 8GB VM, is throwing memory errors

[admin admin]

OK, so my test Jira server is not starting;

I am testing in a VM configured with 8GB of RAM.

I have confirmed that Jira is configured to run with;

JVM_MINIMUM_MEMORY="4g"
JVM_MAXIMUM_MEMORY="6g"

When I review the installation logs;

everything runs as expected until;

2019-06-10 15:19:06,053 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] Running JIRA startup checks.
2019-06-10 15:19:06,053 JIRA-Bootstrap FATAL [c.a.jira.startup.JiraStartupLogger] Startup check failed. JIRA will be locked.

So I then review the application logs;

2019-06-10 15:19:05,871 JIRA-Bootstrap WARN [c.a.jira.health.HealthChecks] JIRA runs in a Java virtual machine (JVM), and this JVM is allocated an amount of memory to enable JIRA to run. These amounts can be manually configured.
- The initial memory allocation pool (-Xms) is recommended to be at least 384m.

2019-06-10 15:19:05,872 JIRA-Bootstrap ERROR [c.a.jira.health.HealthChecks] Your memory allocation pool settings need attention
2019-06-10 15:19:05,872 JIRA-Bootstrap ERROR [c.a.jira.health.HealthChecks] JIRA runs in a Java virtual machine (JVM), and this JVM is allocated an amount of memory to enable JIRA to run. These amounts can be manually configured.
- The maximum memory allocation pool (-Xmx) is required to be at least 786m.

2019-06-10 15:19:06,053 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] Running JIRA startup checks.
2019-06-10 15:19:06,053 JIRA-Bootstrap FATAL [c.a.jira.startup.JiraStartupLogger] Startup check failed. JIRA will be locked.

~ SO, it appears to be halting because it is not operating with enough RAM, but the minimum requisites specified; Xms=384m, and Xmx=786m, are well looked after by the server settings;JVM_MINIMUM_MEMORY="4g" & JVM_MAXIMUM_MEMORY="6g"

So, I am stuck on this, and any assistance would be welcome.

[Nic Brough -Adaptavist- · Accepted solution]

I'd think about brute-force as well - the two JVM parameters you've got there are relatively new, and (almost) substitute the ms/mx pair you mention.  I'd try adding them into setenv.sh as well.  If it then works ok, then at least we know the JVM_.... are not being translated correctly.

I mean something like adding:

CATALINA_OPTS="-Xms384m -Xmx786m  ${CATALINA_OPTS}"

If that changes the error message, I'd then give

CATALINA_OPTS="-Xms4096m -Xmx6144m -XX:+UseG1GC ${CATALINA_OPTS}"

a try, to see if it runs in the enlarged memory space

[admin admin]

Hi Nic,

Thanks for the suggestion;

Adding ' CATALINA_OPTS="-Xms384m -Xmx786m  ${CATALINA_OPTS}" ' to the setenv.sh file does result in no errors being thrown during jira startup, AND the changes are now being correlated in the Jira application/home log as below;

Maximum Allowable Memory : 786MB
Total Memory : 384MB

HOWEVER, the server is still not operating, but now I have no where else to turn, as there are no actionable errors in either the installation or home directory.

[Nic Brough -Adaptavist-]

What do you mean by "not operating"?  Is it running and ignoring the browser?  Is it stopping after launch.  Have you given it enough time to start up?

[admin admin]

The service appears to be started, and no errors are displayed in either Catalina.out, nor atlassian-jira.log, but nothing is coming up at the ip address for the server on port 8080.

We've had this discussion previously if I recall right, in this case however I have not installed tomcat.

[admin admin]

Noting that the server is listening on port 8080... but it appears to be listening on ipv6, I am curious if you know a way to configure the server to listen on 0.0.0.0:8080 as opposed to :::8080.

[admin admin]

[jira@jira bin]$ netstat -tulnp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN -
tcp6 0 0 127.0.0.1:8005 :::* LISTEN 26907/java
tcp6 0 0 :::8080 :::* LISTEN 26907/java
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 ::1:25 :::* LISTEN -
udp 0 0 127.0.0.1:323 0.0.0.0:* -
udp6 0 0 ::1:323 :::* -

[admin admin]

Apologies,
The point I was trying to make was that Jira 'tcp6 0 0 :::8080 :::* LISTEN 26907/java' appears to be listening on ipv6 { :::8080 :::* }, but I use ipv4 {0.0.0.0:8080 0.0.0.0:* ) for our office, so I am curious if/how this can be altered to be backwards compatible with ipv4.

It may be possible that this was also what resulted in the previous issue you helped me with, where I inadvertently installed a subsidiary tomcat as a solution, as looking back that does make sense if the one Jira installs with itself is not IPv4 compatible.

[admin admin]

OK, so I've resolved the memory issue by adding CATALINA_OPTS="-Xms384m -Xmx786m  ${CATALINA_OPTS}" to the setenv.sh file.

The server is still not displaying any output at IP_ADDR:8080 though.

The Jira service is listening on 8080,

bin]$ netstat -tulnp | grep 8080
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::8080 :::* LISTEN 12029/java

bin]$ ps aux | grep 12029
jira 12029 2.7 3.6 2644728 288680 pts/0 SLl 16:56 0:22 //bin/java -Djava.util.logging.config.file=/opt/atlassian/atlassian-jira-software-8.2.1-standalone/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -XX:+UseParallelGC -Xlog:gc*:file=/opt/atlassian/atlassian-jira-software-8.2.1-standalone/logs/atlassian-jira-gc-%t.log:time,uptime:filecount=5,filesize=20M -Xms384m -Xmx786m -Dignore.endorsed.dirs= -classpath /opt/atlassian/atlassian-jira-software-8.2.1-standalone/bin/bootstrap.jar:/opt/atlassian/atlassian-jira-software-8.2.1-standalone/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/atlassian-jira-software-8.2.1-standalone -Dcatalina.home=/opt/atlassian/atlassian-jira-software-8.2.1-standalone -Djava.io.tmpdir=/opt/atlassian/atlassian-jira-software-8.2.1-standalone/temp org.apache.catalina.startup.Bootstrap start

[Alexis Robert]

Hi @admin admin , 

is this a Linux or Windows server ? do you know what file you modified to set the JVM_MINIMUM_MEMORY and JVM_MAXIMUM_MEMORY settings ? 

[admin admin]

Hi Alexis,

Sorry for not adding it in the OP; I added a comment that I was installing from the tar.gz file, so it is Linux, specifically CentOS 7

The file modified to provide the JVM_MINIMUM_MEMORY & JVM_MAXIMUM_MEMORY settings was;

$JIRA_INSTALLATION_DIR/bin/setenv.sh

[Alexis Robert]

Hi @admin admin , 

this is definitely the right file, so your settings should be set in Jira. Can you check the logs in JIRA_INSTALLATION_DIR/logs/catalina.out and look at the logs just after the startup, you should see a summary of the different JVM options there. 

[admin admin]

****************
JIRA starting...
****************

2019-06-10 15:19:05,489 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

___ Environment _____________________________

JIRA Build : 8.2.1#802001-sha1:8875eca5f07af11c7cb5069d6914b23e91407e11
Build Date : Mon May 27 00:00:00 UTC 2019
JIRA Installation Type : Standalone
Application Server : Apache Tomcat/8.5.35 - Servlet API 3.1
Java Version : 11.0.3 - Eclipse OpenJ9
Current Working Directory : /opt/atlassian/atlassian-jira-software-8.2.1-standalone/bin
Maximum Allowable Memory : 1955MB
Total Memory : 43MB
Free Memory : 10MB
Used Memory : 33MB
Memory Pool: class storage : class storage: init = 0(0K) used = 20306088(19830K) committed = 21085032(20590K) max = -1(-1K)
Memory Pool: nursery-survivor : nursery-survivor: init = 1048576(1024K) used = 1310720(1280K) committed = 1310720(1280K) max = 63269888(61787K)
Memory Pool: nursery-allocate : nursery-allocate: init = 1048576(1024K) used = 2511728(2452K) committed = 9306112(9088K) max = 449221632(438693K)
Memory Pool: miscellaneous non-heap storage : miscellaneous non-heap storage: init = 17825792(17408K) used = 5078128(4959K) committed = 22020096(21504K) max = -1(-1K)
Memory Pool: JIT code cache : JIT code cache: init = 268435456(262144K) used = 0(0K) committed = 268435456(262144K) max = 268435456(262144K)
Memory Pool: tenured-SOA : tenured-SOA: init = 5977088(5837K) used = 30976448(30250K) committed = 32935936(32164K) max = 1945911296(1900304K)
Memory Pool: tenured-LOA : tenured-LOA: init = 314368(307K) used = 0(0K) committed = 1732608(1692K) max = 102416384(100016K)
Memory Pool: JIT data cache : JIT data cache: init = 0(0K) used = 2097152(2048K) committed = 2097152(2048K) max = 402653184(393216K)
JVM Input Arguments : -Xoptionsfile=/usr/lib/jvm/adoptopenjdk-11-openj9/lib/options.default -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harmony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry,noLockword=java/lang/invoke/MethodType,noLockword=java/lang/invoke/MethodHandle,noLockword=java/lang/invoke/CollectHandle,noLockword=java/lang/invoke/ConstructorHandle,noLockword=java/lang/invoke/ConvertHandle,noLockword=java/lang/invoke/ArgumentConversionHandle,noLockword=java/lang/invoke/AsTypeHandle,noLockword=java/lang/invoke/ExplicitCastHandle,noLockword=java/lang/invoke/FilterReturnHandle,noLockword=java/lang/invoke/DirectHandle,noLockword=java/lang/invoke/ReceiverBoundHandle,noLockword=java/lang/invoke/DynamicInvokerHandle,noLockword=java/lang/invoke/FieldHandle,noLockword=java/lang/invoke/FieldGetterHandle,noLockword=java/lang/invoke/FieldSetterHandle,noLockword=java/lang/invoke/StaticFieldGetterHandle,noLockword=java/lang/invoke/StaticFieldSetterHandle,noLockword=java/lang/invoke/IndirectHandle,noLockword=java/lang/invoke/InterfaceHandle,noLockword=java/lang/invoke/VirtualHandle,noLockword=java/lang/invoke/PrimitiveHandle,noLockword=java/lang/invoke/InvokeExactHandle,noLockword=java/lang/invoke/InvokeGenericHandle,noLockword=java/lang/invoke/VarargsCollectorHandle,noLockword=java/lang/invoke/ThunkTuple -Xjcl:jclse29 -Dcom.ibm.oti.vm.bootstrap.library.path=/usr/lib/jvm/adoptopenjdk-11-openj9/lib/compressedrefs:/usr/lib/jvm/adoptopenjdk-11-openj9/lib -Dsun.boot.library.path=/usr/lib/jvm/adoptopenjdk-11-openj9/lib/compressedrefs:/usr/lib/jvm/adoptopenjdk-11-openj9/lib -Djava.library.path=/usr/lib/jvm/adoptopenjdk-11-openj9/lib/compressedrefs:/usr/lib/jvm/adoptopenjdk-11-openj9/lib:/usr/lib64:/usr/lib -Djava.home=/usr/lib/jvm/adoptopenjdk-11-openj9 -Duser.dir=/opt/atlassian/atlassian-jira-software-8.2.1-standalone/bin --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=/opt/atlassian/atlassian-jira-software-8.2.1-standalone/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -XX:+UseParallelGC -Xlog:gc*:file=/opt/atlassian/atlassian-jira-software-8.2.1-standalone/logs/atlassian-jira-gc-%t.log:time -Dignore.endorsed.dirs= -Djava.class.path=/opt/atlassian/atlassian-jira-software-8.2.1-standalone/bin/bootstrap.jar:/opt/atlassian/atlassian-jira-software-8.2.1-standalone/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/atlassian-jira-software-8.2.1-standalone -Dcatalina.home=/opt/atlassian/atlassian-jira-software-8.2.1-standalone -Djava.io.tmpdir=/opt/atlassian/atlassian-jira-software-8.2.1-standalone/temp -Dsun.java.command=org.apache.catalina.startup.Bootstrap start -Dsun.java.launcher=SUN_STANDARD -Dsun.java.launcher.pid=7731
Java Compatibility Information : JIRA version = 8.2.1, Java Version = 11.0.3

___ Java System Properties _________________

awt.toolkit : sun.awt.X11.XToolkit
catalina.base : /opt/atlassian/atlassian-jira-software-8.2.1-standalone
catalina.home : /opt/atlassian/atlassian-jira-software-8.2.1-standalone
catalina.useNaming : true
com.ibm.cpu.endian : little
com.ibm.jcl.checkClassPath :
com.ibm.oti.configuration : scar
com.ibm.oti.shared.enabled : false
com.ibm.oti.vm.library.version : 29
com.ibm.system.agent.path : /usr/lib/jvm/adoptopenjdk-11-openj9/lib
com.ibm.util.extralibs.properties :
com.ibm.vm.bitmode : 64
common.loader : "${catalina.base}/lib",
"${catalina.base}/lib/*.jar",
"${catalina.home}/lib",
"${catalina.home}/lib/*.jar"
file.encoding : UTF-8
file.encoding.pkg : sun.io
ibm.signalhandling.rs : false
ibm.signalhandling.sigchain : true
ibm.signalhandling.sigint : true
ibm.system.encoding : UTF-8
ignore.endorsed.dirs :
java.awt.fonts :
java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment
java.awt.printerjob : sun.print.PSPrinterJob
java.class.version : 55.0
java.compiler : j9jit29
java.fullversion : 11.0.3+7
JRE 11 Linux amd64-64-Bit Compressed References 20190417_202 (JIT enabled,
AOT enabled)
OpenJ9 - bad1d4d06
OMR - 4a4278e6
JCL - 5cc996a803 based on jdk-11.0.3+7
java.home : /usr/lib/jvm/adoptopenjdk-11-openj9
java.io.tmpdir : /opt/atlassian/atlassian-jira-software-8.2.1-standalone/temp
java.naming.factory.initial : org.apache.naming.java.javaURLContextFactory
java.naming.factory.url.pkgs : org.apache.naming
java.protocol.handler.pkgs : org.apache.catalina.webresources
java.runtime.name : OpenJDK Runtime Environment
java.runtime.version : 11.0.3+7
java.specification.name : Java Platform API Specification
java.specification.vendor : Oracle Corporation
java.specification.version : 11
java.util.logging.config.file : /opt/atlassian/atlassian-jira-software-8.2.1-standalone/conf/logging.properties
java.util.logging.manager : org.apache.juli.ClassLoaderLogManager
java.vendor : Eclipse OpenJ9
java.vendor.url : http://www.eclipse.org/openj9
java.vendor.version : AdoptOpenJDK
java.version : 11.0.3
java.version.date : 2019-04-16
java.vm.info : JRE 11 Linux amd64-64-Bit Compressed References 20190417_202 (JIT enabled,
AOT enabled)
OpenJ9 - bad1d4d06
OMR - 4a4278e6
JCL - 5cc996a803 based on jdk-11.0.3+7
java.vm.name : Eclipse OpenJ9 VM
java.vm.specification.name : Java Virtual Machine Specification
java.vm.specification.vendor : Oracle Corporation
java.vm.specification.version : 11
java.vm.vendor : Eclipse OpenJ9
java.vm.version : openj9-0.14.0
jdk.extensions.name : Extensions for OpenJDK for Eclipse OpenJ9
jdk.extensions.version : 11.0.3.0
jdk.tls.ephemeralDHKeySize : 2048
org.apache.catalina.security.SecurityListener.UMASK : 0027
os.arch : amd64
os.encoding : UTF-8
os.name : Linux
os.version : 3.10.0-957.12.2.el7.x86_64
package.access : sun.,
org.apache.catalina.,
org.apache.coyote.,
org.apache.jasper.,
org.apache.tomcat.
package.definition : sun.,
java.,
org.apache.catalina.,
org.apache.coyote.,
org.apache.jasper.,
org.apache.naming.,
org.apache.tomcat.
server.loader :
shared.loader :
sun.arch.data.model : 64
sun.boot.library.path : /usr/lib/jvm/adoptopenjdk-11-openj9/lib/compressedrefs:/usr/lib/jvm/adoptopenjdk-11-openj9/lib
sun.cpu.endian : little
sun.io.unicode.encoding : UnicodeLittle
sun.java.command : org.apache.catalina.startup.Bootstrap start
sun.java.launcher : SUN_STANDARD
sun.java.launcher.pid : 7731
sun.jnu.encoding : UTF-8
tomcat.util.buf.StringCache.byte.enabled : true
tomcat.util.scan.StandardJarScanFilter.jarsToScan : log4j-taglib*.jar,
log4j-web*.jar,
log4javascript*.jar,
slf4j-taglib*.jar
tomcat.util.scan.StandardJarScanFilter.jarsToSkip : annotations-api.jar,
ant-junit*.jar,
ant-launcher.jar,
ant.jar,
asm-*.jar,
aspectj*.jar,
bootstrap.jar,
catalina-ant.jar,
catalina-ha.jar,
catalina-jmx-remote.jar,
catalina-storeconfig.jar,
catalina-tribes.jar,
catalina-ws.jar,
catalina.jar,
cglib-*.jar,
cobertura-*.jar,
commons-beanutils*.jar,
commons-codec*.jar,
commons-collections*.jar,
commons-daemon.jar,
commons-dbcp*.jar,
commons-digester*.jar,
commons-fileupload*.jar,
commons-httpclient*.jar,
commons-io*.jar,
commons-lang*.jar,
commons-logging*.jar,
commons-math*.jar,
commons-pool*.jar,
dom4j-*.jar,
easymock-*.jar,
ecj-*.jar,
el-api.jar,
geronimo-spec-jaxrpc*.jar,
h2*.jar,
hamcrest-*.jar,
hibernate*.jar,
httpclient*.jar,
icu4j-*.jar,
jasper-el.jar,
jasper.jar,
jaspic-api.jar,
jaxb-*.jar,
jaxen-*.jar,
jdom-*.jar,
jetty-*.jar,
jmx-tools.jar,
jmx.jar,
jsp-api.jar,
jstl.jar,
jta*.jar,
junit-*.jar,
junit.jar,
log4j*.jar,
mail*.jar,
objenesis-*.jar,
oraclepki.jar,
oro-*.jar,
servlet-api-*.jar,
servlet-api.jar,
slf4j*.jar,
taglibs-standard-spec-*.jar,
tagsoup-*.jar,
tomcat-api.jar,
tomcat-coyote.jar,
tomcat-dbcp.jar,
tomcat-i18n-en.jar,
tomcat-i18n-es.jar,
tomcat-i18n-fr.jar,
tomcat-i18n-ja.jar,
tomcat-i18n-ru.jar,
tomcat-jdbc.jar,
tomcat-jni.jar,
tomcat-juli-adapters.jar,
tomcat-juli.jar,
tomcat-util-scan.jar,
tomcat-util.jar,
tomcat-websocket.jar,
tools.jar,
websocket-api.jar,
wsdl4j*.jar,
xercesImpl.jar,
xml-apis.jar,
xmlParserAPIs-*.jar,
xmlParserAPIs.jar,
xom-*.jar
user.country : US
user.dir : /opt/atlassian/atlassian-jira-software-8.2.1-standalone/bin
user.home : /home/jira
user.language : en
user.name : jira
user.timezone : Etc/UTC
user.variant :

[admin admin]

Hi Alexis,

The resolution suggested below by Nic Brough, resolved the errors in the log regarding memory.

Now however, the server is still not displaying anything and the logs are providing no errors.

To resolve this memory issue the following line was added into the setenv.sh file;

CATALINA_OPTS="-Xms384m -Xmx786m  ${CATALINA_OPTS}"

~Though it can also be modified to your minimum (Xms) and maximum (Xmx) RAM configurations.

[admin admin]

As a note, I am delivering this server with the tar.gz files, and everything else appears to be working as expected, but some additional unforeseen configuration may be necessary?

[admin admin]

I've also attempted starting the server with;

JVM_MINIMUM_MEMORY="4096m"
JVM_MAXIMUM_MEMORY="6144m"

Which did not alter the warnings provided previously.