Confluence eats lots of RAM

[Martin Hynek]

Hello,

I've updated confluence to latest version on my test environment and i was surprised, that even though JVM has alocated 2GB of memory, confluence processes consumes about 13GB!! of RAM. Maybe i didnt understand documentation correctly, but could anyone give me a tip or explanation what should i check or could you confirm that this behavior is standard?

Thanks

[Olaf Brenn]

still the same story, whatever I assign I, confluence is crashing after a while, when I use 6 GB for xms, confluence is using 6,1 GB according to top - and there is the other confluence task running, in melody I see a peak which reaches the limit and therefore crashes the maschine.

I assume that peak is some job - maybe indexing.      

No I try with CATALINA_OPTS="-Xms6144m -Xmx8192m -XX:+UseG1GC ${CATALINA_OPTS}"

My conf had a setting for maxpermsize as well.

Olaf

[geocine]

have you resolved this issue? I am also encountering the same.

[David]

Same Problem here.

Java params:

-Xms3G -Xmx3G

top shows nearly 6GB consumed by both java processes (confluence + synchrony)

Leads regularly to warnings from our monitoring.

After restarting, it seems to be fine for a couple of days, but starts again to grow. We are encountering this issue since we switched to openJDK.

[Olaf Brenn]

today I got as follows, this cannot be normal!

[Olaf Brenn]

today I got this memory consumption

Looks like a not good enough working GC - btw.- the VM has 4 GB RAM

[Olaf Brenn]

there are only 3 users and both use central authentication-method - provided by Jira.

What difference makes the authentication-method?  

[Olaf Brenn]

still the installation is strange. I have pretty much the same installation for 2 companies. Both have appr. the same content. Both differ in RAM consumption as indicated in the graph.

The second has now 4 GB, the first 2 GB RAM.

     

[Björn Rohlén]

This is normal. We have to increase heap-size by a lot to avoid problems, the defaults are no where NEAR enough.

setenv.sh:

CATALINA_OPTS="-Xms6144m -Xmx8192m -XX:+UseG1GC ${CATALINA_OPTS}"

... and top shows:

11255 confluence   20   0 13.314g 8.097g  36856 S   5.6 52.2  17940:25 java

[Olaf Brenn]

thanks, do I really have to  increase to 6/8 GB - today I had memory consumption of 2,8 GB in top. I will increase and try. 

[Björn Rohlén]

No, I suppose you do not "have to", your mileage may vary depending on number of users and what authentication-method you are using. Monitoring and analyzing the application is the only way to see how / if it needs tuning.

[Olaf Brenn]

I installed Javamelody and got the following results. I had to increase memory to 2 GB to cover the memory peaks.There has to be a problem with memory consumtion.  

[Olaf Brenn]

at least it does not work with our setup with the default set of 1 GB RAM. Now it seems to be stable.

[Olaf Brenn]

I don't really care about the consumtion but confluence crashes after appr. 1 h - I have to restart the app. I assume it is a memory issue.

Jira and bitbucket are running fine on the same maschine.

Olaf   

[cloud-operations]

FWIW we see the same thing - Confluence 6.0.4

PID  USER     PR NI VIRT  RES  SHR  S %CPU %MEM TIME+     COMMAND
2837 confluen 20 0  7651m 5.8g 25m  S 74   73.9 92090:48  java  

Java min and max heap size is set to 4G.

So it's using almost 2G on non-heap stuff, which is a huge amount!

[Olaf Brenn]

does anybody know if this issue is solved within the latest version as we are about to update soon?  

[Olaf Brenn]

same problem with my install - works on a 12 GB RAM machine and uses all of the RAM within little time.

On my system there is no bin/setenv.sh.

Even TOP show 96 % load as well as VM proxmox.

System is rather slow and not stable.

I am very curious that the java task with user confluence us running twice - is that correct? Is there any place to see which task are configured by atlassian - I mean ports and services?    

 Olaf

btw. we run jira and conf. on the same server

btw.2 I have 2 different machines with atlassian tools, both behave the same  way  

[Nic Brough -Adaptavist-]

If you don't have a setenv.sh then you are setting the variables needed in the environment.  Atlassian applications generally won't start at all if you don't give them an environment with all the stuff they need, including memory settings.  I'd suggest you take another look at the installation - is it an off-the-shelf install, or have you done something like extracting the application from an install and installing it in another web-server (different version of Tomcat or Glassfish etc)

[Olaf Brenn]

thx, my setenv is located at: /opt/atalassian/confluence/bin/

echo "If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installati$

# set the location of the pid file
if [ -z "$CATALINA_PID" ] ; then
    if [ -n "$CATALINA_BASE" ] ; then
        CATALINA_PID="$CATALINA_BASE"/work/catalina.pid
    elif [ -n "$CATALINA_HOME" ] ; then
        CATALINA_PID="$CATALINA_HOME"/work/catalina.pid
    fi
fi
export CATALINA_PID

PRGDIR=`dirname "$0"`
if [ -z "$CATALINA_BASE" ]; then
  if [ -z "$CATALINA_HOME" ]; then
    LOGBASE=$PRGDIR
    LOGTAIL=..
  else
    LOGBASE=$CATALINA_HOME
    LOGTAIL=.
  fi
else
  LOGBASE=$CATALINA_BASE
  LOGTAIL=.
fi

PUSHED_DIR=`pwd`
cd $LOGBASE
cd $LOGTAIL
LOGBASEABS=`pwd`
cd $PUSHED_DIR

LOGBASEABS=`pwd`
cd $PUSHED_DIR

echo ""
echo "Server startup logs are located in $LOGBASEABS/logs/catalina.out"

# Set the JVM arguments used to start Confluence. For a description of the options, see
# http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
CATALINA_OPTS="-XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution ${CATALINA_OPTS}"
CATALINA_OPTS="-Xloggc:$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:G1ReservePercent=20 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.awt.headless=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.plugins.enable.wait=300 ${CATALINA_OPTS}"
CATALINA_OPTS="-Xms1024m -Xmx1024m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 ${CATALINA_OPTS}"


export CATALINA_OPTS

[Olaf Brenn]

[Olaf Brenn]

so what I see is, that confluence with java is running twice and config says1 GB.

All of the settings are standard.

Olaf 

[Nic Brough -Adaptavist-]

Ah, not quite, if you're on Confluence 6+ then the Confluence user will be running two processes, both java, but one is Confluence, and the other is Synchrony.  "ps -ef | grep confl" will expose that (it's worth checking)

So, the problem now is what makes you think it is consuming 12Gb of RAM, when both processes are (probably) capped at 1Gb?  There is some overhead on top of the heap, as Roman mentioned, but it shouldn't be a lot.

[Steven F Behnke]

There's no way anyone can really answer this question. This doesn't 'sound' standard but it could be well within the settings you've given the JVM.

• How are you determining memory usage?
• What settings have you applied to your JVM?

[Nic Brough -Adaptavist-]

Java puts a hard cap on the RAM used if you've set it correctly.   I suspect when you say "JVM has allocated 2GB", that's not really being applied, so Confluence is quite happily eaten piles of memory and never done any garbage collection to free any.

Have a look at your Confluence's system information page, that has RAM usage data, and can answer Steven's second question (look for Xmx )

[Martin Hynek]

Well for first run of new version i did not change anything except of xms and xmx and set it to 2048m... nothing else changed.. I want to try start without any more changes....

Memory usage read from general info from cnf:

Java VM Memory Stats
Heap memory
  Used memory (22%) Free memory (78%)
   78% Free
Maximum heap size 2048 MB
Total Allocated Memory 2048 MB
Memory used 460 MB
Free allocated memory of 1587 MB
Total free memory 1587 MB

and from linux virtual server top utility, where after start of confluence it consumes 13GB of RAM

Confluence Usage Stats
Total spaces 58
Portals 49
Personal spaces 9
Content (all versions) 73972
Content (current version) 66163
Local users 31166
Local groups 4012

To be honest i'm not sure which setup should be used or how balance it correctly...

I think thats only small confluence, but with lots of users...

[Steven F Behnke]

Can you show us actual text from the bin/setenv.sh file and the actual output from top?

[Nic Brough -Adaptavist-]

The "heap size" in there is recorded as 2Gb, which means there's no way the JVM will consume more than that.  Your settings are all healthy.

So what makes you say it consumes 13Gb, when it can't?

[Martin Hynek]

SetEnv.sh - https://pastebin.com/dukhCEg3

top - https://pastebin.com/X7bppTve

when i check with free -m, i got this..
total used free shared buffers cached
Mem: 15950 15123 827 0 28 944
-/+ buffers/cache: 14150 1800
Swap: 2046 0 2046

all that memory (buffer) allocates during confluence startup

so i thought thats confluences memory... 

[Roman Kirilenko]

Don't worry. Linux shows information about memory usage in a very specific way.  According to your top output, java processes actually use only 3.5Gb of memory (2.8Gb + 707Mb).

There is a second question about 2.8Gb. The answer is simple. You have configured the heap size for Java, but Java also uses metaspace as an additional memory pool that isn't limited by default, so the whole java process consumes 2.8Gb.