Java thread stuck

[George Zheng]

Hello Community team,

I got thread stuck and spike in cpu with following error what does this mean

14-Feb-2022 03:49:24.484 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [ajp-nio-10009-exec-223] (id=[30866]) has been active for [125,035] milliseconds (since [2/14/22 3:47 AM]) to serve the same request for [https://projecttracker.translations.com/browse/ESG-2686] and may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.
 java.lang.Throwable
    at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:213)
    at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
    at org.apache.lucene.store.DataInput.readVInt(DataInput.java:105)
    at org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:181)
    at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:243)
    at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:471)
    at com.atlassian.jira.issue.statistics.util.FieldableDocumentHitCollector.collect(FieldableDocumentHitCollector.java:35)
    at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:281)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:526)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:320)
    at com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:132)
    at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:340)
    at sun.reflect.GeneratedMethodAccessor1917.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:233)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
    at com.onresolve.jira.groovy.jql.AbstractScriptedJqlFunction.search(AbstractScriptedJqlFunction.groovy:155)
    at com.onresolve.jira.groovy.jql.AbstractScriptedJqlFunction$search$6.callCurrent(Unknown Source)
    at com.onresolve.jira.groovy.jql.AbstractScriptedJqlFunction.searchIssuesForDeterminedProjects(AbstractScriptedJqlFunction.groovy:89)
    at com.onresolve.jira.groovy.jql.AbstractScriptedJqlFunction$searchIssuesForDeterminedProjects$1.callCurrent(Unknown Source)
    at com.onresolve.jira.groovy.jql.AbstractScriptedJqlFunction.searchIssuesForDeterminedProjects(AbstractScriptedJqlFunction.groovy:84)
    at com.onresolve.jira.groovy.jql.AbstractScriptedJqlFunction$searchIssuesForDeterminedProjects$0.callCurrent(Unknown Source)
    at com.onresolve.jira.groovy.jql.relations.RelationsOfMatchingQuery.getQuery(RelationsOfMatchingQuery.groovy:250)
    at com.onresolve.jira.groovy.jql.relations.RelationsOfMatchingQuery$getQuery$0.call(Unknown Source)
    at com.onresolve.jira.groovy.jql.ScriptedJqlFunction.getQuery(ScriptedJqlFunction.groovy:128)
    at com.onresolve.jira.groovy.jql.ScriptedJqlFunction$getQuery.call(Unknown Source)
    at com.onresolve.jira.groovy.jql.ScriptedFunctionClauseFactory.getQuery(ScriptedFunctionClauseFactory.groovy:92)
    at com.atlassian.jira.jql.query.ContextAwareQueryVisitor.visit(ContextAwareQueryVisitor.java:97)

[Support Team [Clearvision]]

Hi George,

It's very difficult to identify the course of a stuck thread just by viewing logs.

You would need to take multiple thread dumps at the time the stuck threads and high CPU is occurring.

Guides are available for this e.g.: https://confluence.atlassian.com/adminjiraserver0713/generating-a-thread-dump-964984025.html

If you can also get the command used to output to a file what threads are using CPU that would be useful too, that way you can see which threads are using high CPU and what they are working on at that time. If it tends to be a single event in particular across all thread dumps that would be the first clue.

There is also information about how to analyze thread dumps here: https://confluence.atlassian.com/confkb/how-to-analyze-thread-dumps-788039144.html

Hope that helps.