Advanced search option linkedIssesHasStatuses crushed Jira!

[Michael Thompson]

I am still researching what went wrong here, but wanted to share.

What did you do?

A user issued the following search query in Jira 8.1.0:

project in ("Tech Team Support")
AND issuetype in (sub-task)
And Status in (Resolved)
AND parent in linkedIssuesHasStatuses("Closed")

What happened?

Seconds later, Jira was brought to its knees. I found the following error in catalina.out:

Exception in thread "http-nio-8443-exec-479" java.lang.OutOfMemoryError: GC overhead limit exceeded

Troubleshooting

I brought Jira down cleanly and restarted, but a background re-index failed at 12%....and it took almost 30 minutes to get that far! I will be performing a full re-index after hours to make sure I'm safe.

I am running Jira on CentOS 6.10, with 16GB or RAM and 4 CPUs. Java has been allocated 6 GB of heap space, and usually only uses about 50% of that. I tried reproducing the error in my test environment and observed available memory got from 57% to 3% in about 5 seconds.

Questions

1. Since the query is fine with the first three criteria, has anyone experienced problems using linkedIssuesHasStatuses like we did, where it almost immediately overwhelmed java?
2. Could this sort of failure cause the slow re-indexing I saw?  I just upgraded to Jira 8.1 on May 24 and saw tremendously faster speeds; a background re-index took about an hour to complete. now it appears to be as slow as version 7.12.3, perhaps slower.
3. I thought Jira 8.x was supposed to be impervious to this?!?

Detailed Error Message

31-May-2019 10:29:31.423 WARNING [http-nio-8443-exec-487] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://jira.copyright.com/rest/issueNav/1/issueTable, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
Exception in thread "http-nio-8443-exec-479" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.StringCoding.decode(StringCoding.java:215)
at java.lang.String.<init>(String.java:463)
at java.lang.String.<init>(String.java:515)
at org.apache.lucene.document.DocumentStoredFieldVisitor.stringField(DocumentStoredFieldVisitor.java:75)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.readField(CompressingStoredFieldsReader.java:217)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:590)
at org.apache.lucene.index.CodecReader.document(CodecReader.java:84)
at org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:118)
at org.apache.lucene.index.IndexReader.document(IndexReader.java:373)
at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:332)
at com.atlassian.jira.index.DelegateSearcher.doc(DelegateSearcher.java:108)
at com.atlassian.jira.index.DelegateSearcher.doc(DelegateSearcher.java:108)
at com.atlassian.jira.index.UnmanagedIndexSearcher.doc(UnmanagedIndexSearcher.java:9)
at com.atlassian.jira.index.DelegateSearcher.doc(DelegateSearcher.java:108)
at com.atlassian.jira.index.ManagedIndexSearcher.doc(ManagedIndexSearcher.java:15)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getDocument(LuceneSearchProvider.java:403)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.lambda$search$0(LuceneSearchProvider.java:375)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider$$Lambda$1855/895319834.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.convertTopDocs(LuceneSearchProvider.java:394)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:373)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:135)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:140)
at com.atlassian.jira.bc.issue.search.DefaultSearchService.search(DefaultSearchService.java:118)
at sun.reflect.GeneratedMethodAccessor697.invoke(Unknown Source)