Indexing gets stuck at 5%

[Joel Brass]

While performing a Locked re-index the process gets stuck at 5%. This can last for days.

The JIRA service gets stuck and the machine must be restarted.

Is there any place to look to investigate this error.

Here is the output of the log file.

/secure/admin/jira/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 1% complete. Current index: Issue

2013-08-28 13:06:14,428 IssueIndexer:thread-6 INFO jbrass 786x112x1 1cgreie 67.111.48.66 /secure/admin/jira/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 2% complete. Current index: Issue

2013-08-28 13:06:15,036 IssueIndexer:thread-7 INFO jbrass 786x112x1 1cgreie 67.111.48.66 /secure/admin/jira/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 3% complete. Current index: Issue

2013-08-28 13:06:16,066 IssueIndexer:thread-3 INFO jbrass 786x112x1 1cgreie 67.111.48.66 /secure/admin/jira/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 4% complete. Current index: Issue

2013-08-28 13:06:16,830 IssueIndexer:thread-7 INFO jbrass 786x112x1 1cgreie 67.111.48.66 /secure/admin/jira/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 5% complete. Current index: Issue

[Joel Brass · Accepted solution]

I did set set log levels to DEBUG. This produced much more output that looks like it was pointing to a problem.

Ultimately I fixed the problem by going into "Safe Mode" under the plugins. Once I did this the indexing ran with no problem. I have been re-enabling plugins on an as needed basis and re-running the index with no problems so far.

[Andy Brook [Plugin People]]

Looks ok so far, whats next, what is your CPU activity at this point (idle/full tilt?). If its idle, something broke, go to debug level via System > Logging and Profiling, and repeat to gain more insight, there isnt much to go on in the above.

[Joel Brass]

The CPU activity gets pegged at 95-100% and stays there.

[Andy Brook [Plugin People]]

K, so DEBUG level logging will at least show more (I expect) about what is actually going on. repetetive log messages indicate a looping problem, perhaps due to an exception that is occurring.

If you are have 'dev' ninja skills, enabling JIRA JVM to listen for JPA socket debug connections, you could use an IDE to monitor your reindex (it may be slower!), you could add an java.lang.Exception breakpoint, and see what 'may' be breaking interally.

[JamieA]

Lacking dev ninja skillz, a few stack traces using "jstack pid" might well point to the culprit.