Hundreds of issues reopened after performing a reindex

[Markos Stefanou]

Hello, during the weekend we performed a reindex and today we had many departments reporting numerous (a few hundreds) of closed issues that now appear reopened.

Also, we had a few cases of lost comments after the reindex.

Any thoughts / ideas of how we could resolve the situation?

[Nic Brough -Adaptavist-]

It sounds to me like you have a broken workflow or someone has been modifying the database (you should remove all their access immediately and not let them back until they know what they're doing).  What you are describing suggests that a resolution was set on these issues, the issues indexed properly, and then the resolution removed by a broken workflow or database hack.  A full re-index has "reopened" these issues because they no longer have a resolution.

Re-indexing does NOT write to the database barring a note on the process running, so you did something else that lost the comments. 

I can't really tell you how to fix either of these problems until we know what the root causes are.

[Markos Stefanou]

Thanks for the information, the problem is that we have changed a number of Jira administrators since our last reindex and I lack information in order to deal with the situation efficiently (the last re-index was 6 months ago and I've been working in the company for less than 2 months). 

We are working on fixing them, so far we have located 4,000 such issues. I wish there was an automated solution. It seems that the problem was on one of our most used workflows. I will give a much lower priority on solving the lost comments mystery since they were very few apparently.

[Nic Brough -Adaptavist-]

I would check it carefully for all post-functions (other than the system ones - there's three of those on create, and five on all other transitions).  You'll want to make sure the workflow isn't going to continue to give you broken resolution data.

If you have the Script Runner add-on, there's a built-in script for fixing broken resolutions.

[Markos Stefanou]

We already have the Script Runner installed, I'll check it. (Pretty useful too!)

Sometimes we duplicate workflows and then we delete the previous versions of those workflows. For example, Plain v1 workflow becomes Plain v2 workflow and then after we connect the v2 workflow, we erase  the v1 workflow.

Some issues however have already been resolved through v1 workflow, is it possible that those are the issues that suddenly become open after a reindex?

[Nic Brough -Adaptavist-]

Hmm.  Just to check, are we talking about "reopen" meaning that issues become unresolved, or that they've changed status?

[Markos Stefanou]

They changed status- They were "Done" with resolution "Done" and after the reindex the status changed to "To Do"

[Nic Brough -Adaptavist-]

Ok, it's not the resolution we need to be looking at.

 A re-index simply can't do that, it doesn't write to the database.  So I suspect what has happened is that issues have had their status changed but not been re-indexed.  That can't be done by deleting workflows either - when the issues were migrated to a new workflow, they would have had their status changed to a valid one in the new workflow, and been re-indexed then.

So, there's something somewhere breaking the status, changing it without indexing.  That needs a broken workflow function, a broken script, or database hacks.

[Markos Stefanou]

We've been searching for what the cause could be... The closest to something strange we came up so far is the following.

Even though there was this notice:

"Move function down right before or after firing the event."

The line that contains the Re-indexing "Re-index an issue to keep indexes in sync with the database" appears among the email sending functions. Could this be the case?

[Nic Brough -Adaptavist-]

No, that's fine, the indexing just needs to happen after the status, history and store actions.  Email sends won't affect it, unless they're going wrong and stopping it getting to indexing.  If unsure, move the indexing up above the emails.