Every week we performs a JIRA Background Re-Indexing and most of the time it fails at 97% with the below error.
2018-06-09 16:57:53,355 ISSUE-indexQueue:thread-1 INFO gec-jirasvc [action.admin.index.IndexAdminImpl] Re-indexing is 93% complete. Current index: Issue
2018-06-09 17:02:38,574 ISSUE-indexQueue:thread-1 INFO gec-jirasvc [action.admin.index.IndexAdminImpl] Re-indexing is 94% complete. Current index: Issue
2018-06-09 17:07:17,169 ISSUE-indexQueue:thread-1 INFO gec-jirasvc [action.admin.index.IndexAdminImpl] Re-indexing is 95% complete. Current index: Issue
2018-06-09 17:11:54,462 ISSUE-indexQueue:thread-1 INFO gec-jirasvc [action.admin.index.IndexAdminImpl] Re-indexing is 96% complete. Current index: Issue
2018-06-09 17:16:33,357 ISSUE-indexQueue:thread-1 INFO gec-jirasvc [action.admin.index.IndexAdminImpl] Re-indexing is 97% complete. Current index: Issue
xx.xx.xx.xxx,xx.xx.xx.xx 2018-06-09 17:17:41,441 http-bio-8081-exec-12904 WARN anonymous 1037x11155008x2 - /plugins/servlet/streams [auth.trustedapps.filter.TrustedApplicationFilterAuthenticator] User 'vn0l52t' referenced by trusted application: 'jira:8098344' is not found.
xx.xx.xx.xxx,xx.xx.xx.xx 2018-06-09 17:18:42,044 JiraTaskExectionThread-508 INFO abcde 541x10981514x3 116sphq /secure/admin/IndexReIndex.jspa [jira.issue.index.DefaultIndexManager] 1643701 issues reindexed in the background, in 29826927 millis.
xx.xx.xx.xxx,xx.xx.xx.xx 2018-06-09 17:18:43,893 JiraTaskExectionThread-508 ERROR abcde 541x10981514x3 116sphq /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All In Background FAILED. Indexer: DefaultIndexManager: paths: [/app/jira-home/caches/indexes/comments, /app/jira-home/caches/indexes/issues, /app/jira-home/caches/indexes/changes, /app/jira-home/caches/indexes/worklogs]
java.lang.NullPointerException
at com.atlassian.jira.issue.index.DefaultIssueIndexer$3.consume(DefaultIssueIndexer.java:337)
at com.atlassian.jira.issue.index.DefaultIssueIndexer$3.consume(DefaultIssueIndexer.java:321)
at com.atlassian.jira.util.collect.CollectionUtil.foreach(CollectionUtil.java:44)
at com.atlassian.jira.issue.util.IssueIdsIssueIterable.foreach(IssueIdsIssueIterable.java:27)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.perform(DefaultIssueIndexer.java:320)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.reindexIssues(DefaultIssueIndexer.java:174)
at com.atlassian.jira.index.IssueIndexHelper.fixupConcurrentlyIndexedIssues(IssueIndexHelper.java:117)
at com.atlassian.jira.issue.index.DefaultIndexManager.doBackgroundReindex(DefaultIndexManager.java:1095)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:322)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssuesInBackground(DefaultIndexManager.java:414) <+3> (NativeMethodAccessorImpl.java:62) (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:83)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
at com.sun.proxy.$Proxy15.reIndexIssuesInBackground(Unknown Source)
at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexIssuesInBackground(CompositeIndexLifecycleManager.java:123)
at com.atlassian.jira.web.action.admin.index.ReIndexBackgroundIndexerCommand.doReindex(ReIndexBackgroundIndexerCommand.java:37)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:59)
at com.atlassian.jira.web.action.admin.index.ReIndexBackgroundIndexerCommand.call(ReIndexBackgroundIndexerCommand.java:16)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:24)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:528)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:491)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:254)
at java.lang.Thread.run(Thread.java:745)
xx.xx.xx.xxx,xx.xx.xx.xx 2018-06-09 17:18:43,893 JiraTaskExectionThread-508 INFO abcde 541x10981514x3 116sphq /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing finished
xx.xx.xx.xxx,xx.xx.xx.xx 2018-06-09 17:18:43,893 JiraTaskExectionThread-508 ERROR abcde 541x10981514x3 116sphq /secure/admin/IndexReIndex.jspa [atlassian.jira.task.TaskManagerImpl] Task 'JIRA Indexing' failed.
java.lang.NullPointerException
at com.atlassian.jira.issue.index.DefaultIssueIndexer$3.consume(DefaultIssueIndexer.java:337)
at com.atlassian.jira.issue.index.DefaultIssueIndexer$3.consume(DefaultIssueIndexer.java:321)
at com.atlassian.jira.util.collect.CollectionUtil.foreach(CollectionUtil.java:44)
at com.atlassian.jira.issue.util.IssueIdsIssueIterable.foreach(IssueIdsIssueIterable.java:27)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.perform(DefaultIssueIndexer.java:320)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.reindexIssues(DefaultIssueIndexer.java:174)
at com.atlassian.jira.index.IssueIndexHelper.fixupConcurrentlyIndexedIssues(IssueIndexHelper.java:117)
at com.atlassian.jira.issue.index.DefaultIndexManager.doBackgroundReindex(DefaultIndexManager.java:1095)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:322)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssuesInBackground(DefaultIndexManager.java:414) <+3> (NativeMethodAccessorImpl.java:62) (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:83)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
at com.sun.proxy.$Proxy15.reIndexIssuesInBackground(Unknown Source)
at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexIssuesInBackground(CompositeIndexLifecycleManager.java:123)
at com.atlassian.jira.web.action.admin.index.ReIndexBackgroundIndexerCommand.doReindex(ReIndexBackgroundIndexerCommand.java:37)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:59)
at com.atlassian.jira.web.action.admin.index.ReIndexBackgroundIndexerCommand.call(ReIndexBackgroundIndexerCommand.java:16)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:24)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:528)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:491)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:254)
at java.lang.Thread.run(Thread.java:745)
Kindly let me know the potential cause.
backup the database and execute:
DELETE FROM jiraissue WHERE issuenum IS NULL;
in your db.manager.
@Danyal Iqbal how to perform delete operation in db.manger. please refer if any defined process is there. Also is this the only cause of Re-Indexing failure?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
"db.manager" means "whatever tool it is you would normally use to directly access the database to run SQL queries on it".
I would also seriously question why you are re-indexing every week - that, plus the fact you are getting errors smacks of something very wrong that you should be fixing instead of running regular indexing.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Nic Brough -Adaptavist- we are running Re-Indexing every weekend because we have a very big instance and there are always some modifications happens. To synchronise the data we are required to run Re-Indexing every weekend when there is no load on server.
We are using SQL developer tool to access dB. Will this syntax run on that?
Also is there any pre-built script or service in Script Runner we can utilize to automate our regular weekend Re-Indexing?
Can you please elaborate more on "getting errors smacks of something very wrong that you should be fixing"
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
If background re-indexing has that much of an effect on your server, you should be looking at the Data Centre option. Also, if it is because of config changes, you should really be running a full re-index to capture all change. In fact, a full re-index may fix the problem you are having (unlikely, but it could)
But going back to the orginal question, it looks like you have a null or damaged issue This should never happen, but can do in four broad cases.
Whatever the root cause, you should go find it and fix it.
I would also not rule out "damaged index files", as that could be reading something it can't handle, then not writing it back, leaving a duff record that can't be fixed in background. I would try a full re-index first to rule this out, as that would fix that sort of damage, and then wait to see if that cleans it out.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
we always run full re-index. i hope you are referring to the "background re-index". We also run Lockdown but only once in a month.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
No, you need to run a full re-index, not the background one.
But if you're already doing that regularly (which, again, you shouldn't need to), then you know that it's not fixing the problem. Which means it's not a corrupt index and you really need to find out what is writing bad data.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I agree with Nic, a weekly reindex is not justified even if it improves the performance.
If background re-indexing has that much of an effect on your server, you should be looking at the Data Centre option.
I would say that your instance is undersized for the workload. While migrating to the data option is definitely recommended it might be possible to add resources to your server, optimize garbage collection and follow best practices for large instances to ensure a decent performance with the server version.
How many users do you have?
How big is your data(base) file? I believe you are using an oracle database.
Server specs?
If the database is not corrupt i.e none of the cases mentioned above, an oracle specialist should have a look at the database configuration, the alert/transaction log and the AWS report.
What does this SQL say:
alter index validate structure
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Danyal Iqbal Yes we are using Oracle, Yes we'r in planning to move cloud where we'll not be facing space issues. However thanks all of you for your inputs.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Manu Mishra Try the resolution related to jira-issue, as i can see from the Log, the error abcde with a Null-Pointer Exception, so it could be that there is a null issue num entry in the jiraissue table.
This could fix your problem.
https://confluence.atlassian.com/jirakb/indexing-fails-with-reindex-all-failed-error-429917142.html
Best!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Moses Thomas I have already applied the fix mentioned in KB article. Though I forgot to mention in description. I am still facing the same error.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Diagnosis
SELECT id, issuenum, project FROM jiraissue WHERE issuenum is NULL;
Possible Root Cause
Possible solution.
If the query from the diagnosis returns any results, run the following to remove the NULL entries:
DELETE FROM jiraissue WHERE issuenum IS NULL;
Have you tried this one from your data base server ? make sure you backup you Jira database when performing the delete query in-case.
Best
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Moses Thomas we always check the integrity before running re-index. However we still gets the error after validating integrity. I not sure what is conflicting.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Manu MishraHave you tried what i have sent to you from your database server/SQL developer tool , if not contact your database administrator to try resolution i have mention above. this issue is clearly related to solution mentioned.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Moses Thomas If you are referring to the syntax.
SELECT id, issuenum, project FROM jiraissue WHERE issuenum is NULL;
I have already tried this fix but no luck.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Manu MishraDid you get any result when you tried this ?
if you got the perform this
DELETE FROM jiraissue WHERE issuenum IS NULL;
if you did not get any result from the Select query,
the issue could be something else.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.