deleted issue during bulk change and lucene indexing error

[JINWON CHOI]

I have a problem with Lucene indexing, I've tried to solve it but I couldn't.
please let me know how to solve this problem. 

Jira server version: 7.7.1

an issue was deleted during bulk change and then there is a Lucene indexing error.

I found some garbages in databases. I think after deleting the issue, it was updated.

 - jiraissue table (NULL without ID, updated column)
 - changeitem table (changed history field value)

so, I've tried to solve this problem, but I couldn't.

- Background re-index 
- delete cache index file and restart JIRA server
- delete issue using rest api →403, 500 error
- delete issue using url (/secure/DeleteIssue!default.jspa?id=457887) 

Technical details

Log's referral number: 06962041-07f5-442b-b66a-b591d8c2070d

Cause

Referer URL: Unknown

java.lang.IllegalArgumentException: The Project argument and its backing generic value must not be null

Technical details

Log's referral number: 06962041-07f5-442b-b66a-b591d8c2070d

Cause

Referer URL: Unknown

java.lang.IllegalArgumentException: The Project argument and its backing generic value must not be null

java.lang.IllegalArgumentException: The Project argument and its backing generic value must not be null
	at com.atlassian.jira.security.DefaultPermissionManager.doProjectPermissionCheck(DefaultPermissionManager.java:174) [classes/:?]
	at com.atlassian.jira.security.DefaultPermissionManager.doIssuePermissionCheck(DefaultPermissionManager.java:142) [classes/:?]
	at com.atlassian.jira.security.DefaultPermissionManager.doIssuePermissionCheck(DefaultPermissionManager.java:134) [classes/:?]
	at com.atlassian.jira.security.DefaultPermissionManager.hasPermission(DefaultPermissionManager.java:86) [classes/:?]
	at com.atlassian.jira.security.WorkflowBasedPermissionManager.hasPermission(WorkflowBasedPermissionManager.java:82) [classes/:?]
	at com.atlassian.jira.security.ApplicationRequiredPermissionManager.lambda$hasPermission$1(ApplicationRequiredPermissionManager.java:76) [classes/:?]
	at com.atlassian.jira.security.ApplicationRequiredPermissionManager.checkUserHasApplicationOrFalse(ApplicationRequiredPermissionManager.java:182) [classes/:?]
	at com.atlassian.jira.security.ApplicationRequiredPermissionManager.hasPermission(ApplicationRequiredPermissionManager.java:76) [classes/:?]
	at com.atlassian.jira.security.PublicAccessPermissionManager.lambda$hasPermission$1(PublicAccessPermissionManager.java:80) [classes/:?]
	at com.atlassian.jira.security.PublicAccessPermissionManager.checkPublicAccessEnabledOrDelegate(PublicAccessPermissionManager.java:188) [classes/:?]
	at com.atlassian.jira.security.PublicAccessPermissionManager.hasPermission(PublicAccessPermissionManager.java:80) [classes/:?]
	at sun.reflect.GeneratedMethodAccessor441.invoke(Unknown Source) [?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [?:1.8.0_102]
	at java.lang.reflect.Method.invoke(Method.java:498) [?:1.8.0_102]
	at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:22) [classes/:?]
	at com.sun.proxy.$Proxy11.hasPermission(Unknown Source) [?:?]
	at com.atlassian.jira.bc.issue.DefaultIssueService.hasPermissionToView(DefaultIssueService.java:1048) [classes/:?]
	at com.atlassian.jira.bc.issue.DefaultIssueService.getIssue(DefaultIssueService.java:1095) [classes/:?]
	at com.atlassian.jira.bc.issue.DefaultIssueService.getIssue(DefaultIssueService.java:172) [classes/:?]
	at com.atlassian.jira.web.action.issue.DeleteIssue.doDefault(DeleteIssue.java:63) [classes/:?]
	at sun.reflect.GeneratedMethodAccessor38297.invoke(Unknown Source) [?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [?:1.8.0_102]
	at java.lang.reflect.Method.invoke(Method.java:498) [?:1.8.0_102]
	at webwork.util.InjectionUtils$DefaultInjectionImpl.invoke(InjectionUtils.java:70) [webwork-1.4-atlassian-30.jar:?]
	at webwork.util.InjectionUtils.invoke(InjectionUtils.java:56) [webwork-1.4-atlassian-30.jar:?]
	at webwork.action.ActionSupport.invokeCommand(ActionSupport.java:438) [webwork-1.4-atlassian-30.jar:?]
	at webwork.action.ActionSupport.execute(ActionSupport.java:157) [webwork-1.4-atlassian-30.jar:?]
	at com.atlassian.jira.action.JiraActionSupport.execute(JiraActionSupport.java:63) [jira-api-7.7.1.jar:?]
	at webwork.interceptor.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:39) [webwork-1.4-atlassian-30.jar:?]
	at webwork.interceptor.NestedInterceptorChain.proceed(NestedInterceptorChain.java:31) [webwork-1.4-atlassian-30.jar:?]
	at webwork.interceptor.ChainedInterceptor.intercept(ChainedInterceptor.java:16) [webwork-1.4-atlassian-30.jar:?]
	at webwork.interceptor.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:35) [webwork-1.4-atlassian-30.jar:?]
	at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:225) [webwork-1.4-atlassian-30.jar:?]
	at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:154) [webwork-1.4-atlassian-30.jar:?]
	at com.atlassian.jira.web.dispatcher.JiraWebworkActionDispatcher.service(JiraWebworkActionDispatcher.java:138) [classes/:?]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [servlet-api.jar:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) [catalina.jar:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [catalina.jar:8.5.6]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [catalina.jar:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [catalina.jar:8.5.6]
	at com.atlassian.jira.web.filters.JiraLastFilter.lambda$doFilter$0(JiraLastFilter.java:39) [classes/:?]
	at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:74) [classes/:?]
	at com.atlassian.jira.web.filters.JiraLastFilter.doFilter(JiraLastFilter.java:36) [classes/:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [catalina.jar:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [catalina.jar:8.5.6]
	at com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37) [atlassian-core-5.0.8.jar:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [catalina.jar:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [catalina.jar:8.5.6]
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39) [atlassian-plugins-servlet-4.5.0.jar:?]
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) [atlassian-plugins-servlet-4.5.0.jar:?]
	at com.atlassian.jira.onboarding.postsetup.ui.PostSetupAnnouncementsFilter.doFilter(PostSetupAnnouncementsFilter.java:51) [?:?]
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) [atlassian-plugins-servlet-4.5.0.jar:?]
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) [atlassian-plugins-servlet-4.5.0.jar:?]
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) [atlassian-plugins-servlet-4.5.0.jar:?]
	at com.atlassian.jira.tzdetect.IncludeResourcesFilter.doFilter(IncludeResourcesFilter.java:77) [?:?]
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) [atlassian-plugins-servlet-4.5.0.jar:?]
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) [atlassian-plugins-servlet-4.5.0.jar:?]
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) [atlassian-plugins-servlet-4.5.0.jar:?]
	at com.atlassian.jira.baseurl.IncludeResourcesFilter.doFilter(IncludeResourcesFilter.java:40) [?:?]
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) [atlassian-core-5.0.8.jar:?]

[JINWON CHOI]

replied

[Daniel Ebers]

Hi @JINWON CHOI

I am trying to follow - but one question.
Did the error message appear before or after the fix you applied?

In case you haven't seen the knowledge base entry - I just found the following, in case it helps?
https://confluence.atlassian.com/jirakb/jira-functionality-fails-due-to-the-project-argument-and-its-backing-generic-value-must-not-be-null-error-235667577.html

Could you please confirm the fixes mentioned in the article have already been applied by you?
In first step I'd rather recommend not to do further fixes but instead to assess and understand the error thrown first as a whole.

Regards,
Daniel

[JINWON CHOI]

Thanks for your support. 
however, the article is about project error but I need for indexing or issue error.

According to catalina log and jiraissue table, I could guess the deleted issue's key and ID but there are NULL values except ID and updated column in jiraissue table and the updated value is the same as lucene error result time. (12/3/20 2:50 PM)

In addition, I could found change history in changegroup and chageitem table.

Is there any tables and values in database to check more?
Is there any article or documentation for similar situation.

I must to solve this error, please give me some information.

Regards

jinwone choi

[Daniel Ebers]

Well, see, the error message you are getting

The Project argument and its backing generic value must not be null

at least corresponds 1:1 with the knowledge base article.
In case you want to solve the problem it may be worth having a second look.

To your newer question:

For a safe way to look into database trouble I would suggest using the Integrity Checker:
https://confluence.atlassian.com/adminjiraserver/using-the-database-integrity-checker-938847667.html

After all the question is why the database got in this state (assuming there is something broken, from Community view this is not possible to say definitely).
In case there were manual interventions there is some possibility that the current state is not revertable.

[JINWON CHOI]

Thank you for your answer. But it seems different from my problem.

Let me change the question.

Q. What are the DB tables that will be updated when the issue is change?

[Daniel Ebers]

Let me cite some follow leaders here from Community: this information would take you nowhere, database is very complex and there is a huge chance that you destroy more than you repair. Don't take it personal - it is just that approaching the problem from within this angle helped in the rarest cases.

The database schema is documented, of course you can have a deep dive if you want:
https://developer.atlassian.com/server/jira/platform/database-schema/

But understanding, and foremost in a way it will fix the error (given the assumption there IS something wrong with database) will be a tricky thing, I suppose.

[JINWON CHOI]

Thank for your answer.

I've finally deleted jiraissue which is null.

It is related to following url : https://jira.atlassian.com/browse/JSWSERVER-16473

I think you should consider upgrade.

[Daniel Ebers]

I see, this is one of the rare cases when a careful change directly in the database could make sense.
Important would be to know what lead to the inconsistency but this will be hard to determine.

An upgrade to a more recent version is always a good idea.

Glad you fixed it for now.