Experiencing OoM errors with Perm Gen

[Chris Solgat]

We enabled class loading in the logs on our Test environment.  We only had one candidate that was a huge standout.

[Loaded org.codehaus.groovy.jsr223.GroovyScriptEngineFactory from __JVM_DefineClass__]

We had over 9000 entries from our 2 hour test yesterday.  Today we are running a 6 hour test and 2.5 hours into it there have been over 16,500 entries.  Yesterday's test created and closed 2000 issues, which leads us to believe that this class is being loaded on nearly every script call.

My question is, why is this class being loaded on every script call but never being released?  All of the other entries that I came across that load hundreds of entries has a unique ID, I'm assuming so it can be released later.  Should this class have something similar?

[MattS]

I would upgrade script runner and test again. And also force gc and see if the class objects get collected during that

[Chris Solgat]

I cannot upgrade script runner until Jira is upgraded. I may be able to try the forced gc.

[JamieA]

Can you provide a lot more information, such as all the versions concerned, ie java, jira, plugin. What options are you using to generate this information.

What you do you have perm gen max set to? 

I've done a work on this before and concluded that there is no issue. Reluctant to do this again. Have you checked answers for previous posts on this topic? There should be one from me that has a reasonable amount of detail. 

[Chris Solgat]

Java - 1.7.0_45

JIRA - 6.1.6

Script Runner - 2.1.16

Options - XX:+TraceClassLoading -XX:+TraceClassUnloading 

Perm Gen Max - 384M

I have done a little bit of looking around Answers, but mostly related to perm gen, not necessarily this class in particular.  Does the post you mentioned reference this class in particular?  If not, can you think of a key word/phrase that may help me locate it?

I was also using Plumbr JVM monitor while running the load test.  We had a steady increase in the perm gen at the rate of around 1mb every 2 minutes.  Like I said, this was the only class that was showing up consistently at the time, so I don't have many other options to look into.

[JamieA]

That class is not in use in the plugin in 3.x. There are other plugins that use groovy, perhaps it's one of those? Can you verify you get the same behaviour with script runner disabled? How are you using script runner? https://answers.atlassian.com/questions/145473/permgens-code-cache-memory-pool-constantly-at-98 https://answers.atlassian.com/questions/145062/permgen-problems-with-scriptrunner (out of date).

[Chris Solgat]

You say the class is not in use in plugin 3.x. Do you mean Script Runner 3.x? We're not currently using Script Runner 3.x, we are using 2.1.16. As for other plugins using Groovy, here's what I found - Binary file ./.osgi-plugins/transformed-plugins/plugin_9120559714615288802_groovyrunner-2.1.16_1394919097000.jar matches Binary file ./.osgi-plugins/transformed-plugins/plugin.9220850297756612085.jira-misc-workflow-extensions-3.3.0_1394920830000.jar matches Binary file ./.osgi-plugins/felix/felix-cache/bundle141/version0.0/bundle.jar-embedded/META-INF/lib/groovy-all-1.8.5-scriptrunner.jar matches Binary file ./.osgi-plugins/felix/felix-cache/bundle141/version0.0/bundle.jar matches Binary file ./.osgi-plugins/felix/felix-cache/bundle138/version0.0/bundle.jar matches After looking through the two answers you provided links for, I looked for which groovy versions were being run. GroovyRunner and bundle141 are both using 1.8.5. The other 2 are using (I think) version 2.1.6. I found the release information under META-INF/groovy-release-info.properties. As for what script runner is being used for, it is being used for several things. The project in question is mostly using it for some Validators and more than a few Post Functions. In looking through pieces of the log, the GroovyScriptEngineFactory looks like it is being called on every script. I looked through the workflow, and the Misc Workflow Extension is only being used twice and Script Runner is being used around 4-6 times (depending on which path it takes through the workflow). This again has some consistency with the numbers I mentioned previously. During our load test, we created and closed 2000 issues and had 9000 loads. That of course is just over a 4-1 ratio, which follows the Script Runner calls. I have not been able to run the load test without script runner, but am willing to try. I will try to coordinate this test hopefully for some time this week, but have no guarantees with the holidays.

[JamieA]

Can you try this: https://jamieechlin.atlassian.net/browse/GRV-206?focusedCommentId=67277&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-67277 and make sure it returns 1.8.5.

[Chris Solgat]

On the secure/admin/groovy/GroovyRunner.jspa screen, I ran groovy.lang.GroovySystem.getVersion() and it does return 1.8.5. I also ran the this.class.name and it always returns the same.