Performance Issues - Slow loading batch.css

[Daniel Bergamin]

Hi all! Had some reports of slow page loads, and, sure enough, scanning the tomcat access log shows several entries across the day like the following:

URL	Response Code	Bytes Sent	Time
GET /jira/s/d41d8cd98f00b204e9800998ecf8427e/en_AUnqmflm-1988229788/6159/7/09d7cf904052d036ab09c7074fd2e10e/_/download/contextbatch/css/greenhopper-rapid-non-gadget,atl.general,jira.global,gh-rapid-config/batch.css HTTP/1.1	200	72552	172219
GET /jira/s/d41d8cd98f00b204e9800998ecf8427e/en_AUnqmflm-1988229788/6159/7/09d7cf904052d036ab09c7074fd2e10e/_/download/contextbatch/css/gh-rapid-config,greenhopper-rapid-non-gadget,atl.general,jira.global/batch.css HTTP/1.1	200	71753	162733
GET /jira/s/d41d8cd98f00b204e9800998ecf8427e/en_AUnqmflm-1988229788/6159/7/81e09bd7dab1aeeee1270f7f3bb998eb/_/download/contextbatch/css/jira.admin.conf,jira.admin,jira.global,atl.admin/batch.css HTTP/1.1	200	67087	154605
GET /jira/s/d41d8cd98f00b204e9800998ecf8427e/en_AUnqmflm-1988229788/6159/7/81e09bd7dab1aeeee1270f7f3bb998eb/_/download/contextbatch/css/jira.admin.conf/batch.css HTTP/1.1	200	55572	142327
GET /jira/s/d41d8cd98f00b204e9800998ecf8427e/en_AUnqmflm-1988229788/6159/7/8416bb3c6914b4fee5f793acda2e8edd/_/download/contextbatch/css/greenhopper-rapid-non-gadget,atl.general,jira.global/batch.css HTTP/1.1	200	48925	133934
GET /jira/s/d41d8cd98f00b204e9800998ecf8427e/en_AUnqmflm-1988229788/6159/7/d3ab3c91dd139ba82a2d1dddec04d264/_/download/contextbatch/css/jira.global/batch.css HTTP/1.1	200	38563	116336

As you can see above (sorry about the mangled formatting mess), these requests are taking from 48 seconds to ~3 minutes to deliver. Any ideas as to what could be causing these slow loading entries? All of the real slow ones seem to be from the batch css, and I'm not too sure what is going on.

The bulk of the batch CSS request has been responded to in normal times (~100ms). I've had a look for services being kicked off by Quartz which could slow everything down, but cannot find anything.

Thanks for any assistance!

[Tansu Kahyaoglu · Accepted solution]

Take a few thread dump when system goes slow, probably you see bunch of

at org.mozilla.javascript.ScriptRuntime.nameOrFunction(ScriptRuntime.java:1741)
	at org.mozilla.javascript.ScriptRuntime.getNameFunctionAndThis(ScriptRuntime.java:2188)
	at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:61)
	at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_41(less-concat.js:764)
	at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)
	at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
	at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261)
	at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js)
	at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63)
	at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_55(less-concat.js:1115)
	at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)
	at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)

on locked threads. If it is, then visit the issue (https://jira.atlassian.com/browse/JRA-37337) , and vote it :)

[Christian Czaia [Decadis AG]]

Just a shot in the dark, but are you using scripted fields?

Cheers

[Daniel Bergamin]

Hi Christian,

Yes, scripted fields are in use. Do you think this is connected somehow?

[Christian Czaia [Decadis AG]]

Hey, maybe. Depends on the fields :-) If you have very complex scripts that possibly scan through all of the existing issues it will definitley slow down your system...

[Erik Saline [BlackPearl PDM]]

I have a customer with the same problem, they have EazyBI and JIRA Agile installed. I have installed JIRA Agile elsewhere but have not seen this issue.

[Daniel Bergamin]

Hi Ben, if you haven't already, recommending checking Tansu's answer below - I believe this is the problem we were experiencing.

[Benjamin Copeland]

Hey Daniel,

I am running EasyBI and Agile,

We have apache redirect rule for the port forwarding. I disabled gzip on apache, but hasn't changed much.

[Daniel Bergamin]

Hey Ben, can I ask if you are running any or all of the following plugins? ScriptRunner, Behaviours, EazyBI or JIRA Agile.

It might be nothing, but some commonalities here might give me good reason to take a closer look at these plugins.

[Benjamin Copeland]

This is affecting our jira server too. We have 1,202,771ms and 1,085,499 meantime loading these files using JavaMelody 1.50.0. Would be good to have a answer.

[Daniel Bergamin]

I'm hoping that someone will have seen this before and know of the issue - am not expecting anyone to run through a logical analysis of the cause based on the very limited information above.