Hi Pete,
Thanks for the log files, I took a look through these and I found a few things that we probably need to address here. In this specific reply, I'm focusing on the 7.13.1 support zip file you sent:
First there are a number of instance of "FAILED PLUGIN REPORT" showing both failed plugins and other plugin artifacts that can't be resolved. Failed to load:
/srv/jira/plugins/installed-plugins/hipchat-for-jira-plugin-7.8.29.jar
^ this is an old bundled plugin for Jira that isn't bundled any longer in Jira 7.13.1. It should be probably be removed.
But then the logs go on about other plugin problems at startup:
20 plugins are unaccounted for.
Unaccounted for plugins load as artifacts but fail to resolve into full plugins.
/srv/jira/plugins/installed-plugins/jira-software-language-pack-zh_CN-7.13.1.v20190129001300.jar
^ these are mostly language packs, but there are also a few bundled system plugins here too, such as:
/opt/atlassian-jira-software-7.13.1-standalone/atlassian-jira/WEB-INF/atlassian-bundled-plugins/jira-importers-redmine-plugin-2.1.9.jar
/opt/atlassian-jira-software-7.13.1-standalone/atlassian-jira/WEB-INF/atlassian-bundled-plugins/jira-importers-bitbucket-plugin-1.0.19.jar
/opt/atlassian-jira-software-7.13.1-standalone/atlassian-jira/WEB-INF/atlassian-bundled-plugins/jira-importers-github-plugin-3.0.3.jar
/opt/atlassian-jira-software-7.13.1-standalone/atlassian-jira/WEB-INF/atlassian-bundled-plugins/jira-ical-feed-1.4.7.jar
^ some of these Jira might be able to run without, but others could actually be required for Jira to work.
Immediately after that report is this error:
2019-02-13 12:41:47,123 ForkJoinPool.commonPool-worker-1 ERROR [c.a.j.cache.soy.SoyCacheWarmer] Caught exception while compiling soy template
com.atlassian.soy.renderer.SoyException: Unable to compile Soy templates at: com.atlassian.jira.jira-issue-nav-plugin:issuenav-common
Which is more confirmation to me that we have plugins failing to load that Jira needs. You can find very similar errors in the KBs of:
When Jira has plugins that fail to load a number of times, it can set a flag in the database to prevent trying to load these again. That mechanism usually helps to keep 3rd party plugins from completely crippling Jira, however in your case, I'm suspecting that it might have actually disabled system plugins that Jira genuinely requires to run correctly.
It looks like your installation also has a number of old plugin versions. This can happen over time as it looks like you have upgraded this installation many times since version 4.0. Some of these plugins could be benign, but others are known to cause problems in the newer Jira versions.
My suggested approach here is to try to address all the possible causes for this plugin loading failures at once. What I would recommend here would be increase the Xmx in Jira to 2048m up from the default you have of 768m, clear the user plugins from being loaded, and then clear out via SQL the flags Jira might have set to disable specific plugins. Specifically try these steps here:
- Stop Jira
- Increase the memory Jira can use to 2g or 2048m per Increasing memory in Jira
- Go into your /srv/jira/plugins/installed-plugins/ I recommend moving all the contents of this directory (but leave the directory itself) into a temp directory outside your $JIRAHOME directory for the time being
- In the SQL database for Jira, run the command
truncate table pluginstate;
This will remove all entries from that specific table in the database that might have been disabling system plugins from loading at startup. Since we have also removed all the user plugins into the temp directory (in step #3), Jira should then be able to load all the system plugins needed to startup correctly.
- Start Jira again
- Then use the UPM (universal plugin manager) in Jira in order to reinstall the plugins you need from our marketplace. Using the UPM to do this will help prevent you from installing plugins you don't need and only will install compatible plugins with your Jira version as found in Marketplace listings. The only problem with this is that your Jira server needs to access the internet to reach out marketplace site. You can also reinstall Jira Service Desk or Software by going to System -> Applications -> Licenses and Versions. There should be the ability to install the corresponding version of these 'apps' into your Jira site.
- You will probably be prompted to reindex Jira after installing all these plugins again. You should be ok to install all the plugins and then just reindex once.
Once you do those steps, I would expect Jira to be running considerably better here by loading the needed plugins, not loading the old relic plugins, and it should have more than sufficient memory give the size of your Jira data.
If not, I'd be interested to gather another support zip, and if you could, please include the application.xml in addition to the logs? It provides some more environmental details that can help us to understand if the system has appropriate resources like number of CPU Cores, amount of system memory, etc.
Please try these steps and let me know the results.
Andy
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.