tomcat6.exe periodically hogging the CPU at 99% ('apache tomcat confluence' service)

[Linda Petty]

I upgraded to confluence 4.3.2 on 16NOV the upgrade went fine,

but about an 2hrs after the upgrade we found tomcat6.exe was using

99% of the CPU and had been doing so for about 30 mins or more.

I restarted 'apache tomcat confluence' service and this solved the

problem for a while, but soon after I found it running at 99% again,

this cleared itself after 20 mins.

Since then we have had periods of 20mins or more when tomcat uses 99%

of CPU although confluence has been running OK much of the time.

periods of tomcat6.exe using 99% of CPU since then

45 mins 16.45 16nov

20 mins 16.00 16Nov

20 mins 02.30 17nov(sat)

20 mins 17.40 19Nov

20 mins 12.10 20Nov

20 mins 13.50 20Nov

30 mins 14.30 20Nov

You can see they are getting more frequent

and this affects performance.

Confluence runs on windows server 2003 with JIRA running on the same server.

Any suggestions what we can do to avoid this happening?

We have the service set to restart every night.

[Linda Petty · Accepted solution]

We upgraded Java and increased the max memory assigned to the 'Apache tomcat confluence' service as described here https://confluence.atlassian.com/display/CONFKB/How+to+Fix+Out+of+Memory+Errors+by+Increasing+Available+Memory

[Linda Petty]

it was fine for several days after we made those changes, but we did get another peak this morning which has since resolved itself.

[Daniel Borcherding]

Another possible cause could be that you are pulling in an external users from a directory like LDAP on a given interval. This is a less likely cause because your cpu spike interval is non regular. Our connectors can be configured to perform a full sync from an external auth source on a given internval.

[Matthew J. Horn]

To me, it sounds like Tomcat's garbage collection is not working well. I would look for tips on improving it. Here are a couple links:

http://www.theserverside.com/discussions/thread.tss?thread_id=63241

http://stackoverflow.com/questions/8916002/tomcat-garbage-collecting-frequency

[GuilhermeA]

Hello Yeyeo,

Based on past experiences, I've seen the environment be the problem together with Tomcat. What does it mean? That somehow the build of your OS plus Tomcat, is giving your system/tomcat a bad time.

Usually this is resolved moving to another box, which I don't know if its possible, however if you are running on a VM, changing the OS to another would work most of the times, and in other ocasions you can change the Bundled Tomcat to another version.

As tomcat relies on Java, it could be caused by the JDK as well.

Hope this helps you!

Cheers,

Guilherme

[Radu Dumitriu]

1/ Are you sure your confluence is not hit by users during those periods ? In my experience people are coming in "packs" on one site.

2/ kill -3 to get the thread dump (on Windows it's jstack you need to use), check what's that one doing there.

"this cleared itself after 20 mins" => it's not a while(1) ... but you can suspect the upgraded Confluence or the upgraded plugins. The second point above may give a hint.