User installed plug-ins disappeared after JIRA on-premise server restart

[Jason Huntowski]

Recently, we encountered an issue where after a series of server restarts, all of our user installed plug-ins and the service desk application disappeared from our Jira server instance. We are not quite sure what caused the series of server restarts, but because we do not run the administration on our servers, that is less of a mystery than all of our plug-ins just disappearing.

Ultimately, we had to manually reinstall every plug-in...luckily, we had a list. But we did not have time to troubleshoot the issue trying to determine what really happened. Has anyone ever experienced anything like this before? Did you ever find the root cause of it? Could it be perceivable that during the series of restarts one of the .xml files became corrupt causing Jira to lose sight of it's add-ons?

Any info would be greatly appreciated.

Thanks!

[Thomas Vu]

If anybody else runs into this, we have jira hosted in AWS with things like plugins, logs, and attachments on EFS.  Upon server restart, the EFS needed to be remounted.  

Good luck out there.

[Nic Brough -Adaptavist-]

If you hard kill a Jira, then yes, you can damage files or data that might cause an add-on to disappear.  It is however far more likely that someone simply removed them, either from admin or from the disk.

I think you need to get the administrators to find out exactly why there are restarts happening though.

[Jason Huntowski]

Hey Nic,

I don't disagree with you. We are currently trying to find out why the server was restarted so many times. I have been running this particular instance of Jira for over 4 years now and I have never seen anything like it. Just out of curiosity, if someone wanted to remove the plug-ins from the disk, what would the critical file or folder location be? I tried to find where the actual jar and obr files were stored, but could not find them. I thought they would be in the plugins folder with the osgi cache and stuff, but they weren't. Then I was looking around web-inf but got lost and had to start reinstalling to get my instance back up.

Thanks,
Jason

[Nic Brough -Adaptavist-]

You were close, they're all under <jira home>/plugins

One other thing that I remember once.  This caused Jira to need a restart and some add-ons had gone AWOL - disk space.  (Sort of.  It was actually that the disk was out of inodes rather than physical space, but it looks the same as running out of space)

[Jason Huntowski]

Nic,

Thanks for the info. This is a lot to consider. I have some crazy stuff going on actually. Check this out. So, I have a running production instance of Jira. It's a VM on a physical server at my company and is running jira version 7.3.2. I also have a UAT environment running on a VM on a cloud based server (still an on-premise version of Jira) and it's running jira version 7.7.0.

I noticed that the production instance has the plug-in files located at:

Atlassian->Application Data->JIRA->plugins->installed-plugins

These were not there when I was "missing" my plugins, but are there now after I reinstalled.

However, in the UAT environment, none of the plugin files are listed there, but are all still installed in the instance. Why would the plugin files show in this folder for one instance, but not the other? Is it the jira version? Maybe the location where plug-ins are kept has changed between 7.3.2 to 7.7.0?

The fact that the plugins aren't there in my UAT environment caused me some confusion when I tried finding the plugin files to see if they were installed.

In my opinion, it almost seems like someone may have thought they were deleting the osgi cache folders to remove the plugins cache and accidentally removed the installed plugins folder? Would that cause this issue?

Thanks,

Jason

[Vickey Palzor Lepcha]

When you say disappeared - does it mean that it was disabled on the add-ins admin page or were all the jar files wiped out from backend as well - say linux or windows server backend.

[Jason Huntowski]

Hi Vickey,

No, they were not disabled, they were gone. Our list of over 20 plug-ins in the user installed section of manage add-ons was down to 4. I could not locate the jar or obr files on the server, not that I know where they are kept anyway, and the system wasn't able to find them either. The fact that service desk was gone as well was a bit daunting as I am not sure, but it would seem ofdd to me that service desk would be kept in the same place as the other add-ons. Maybe it is, but...

Jason

[Vickey Palzor Lepcha]

If it disappeared just from the add-on list of manage add-ons , it could be a JIRA issue. But if the plugins are lost from the backend - it looks more like an issue or an action outside JIRA - maybe somebody tampered with the installations or something to do with the host OS.

[Vickey Palzor Lepcha]

You will find your plugins both in home and application directory depending on the type of plugins that you are looking for .

Say under application directory - the path is ;

.../atlassian-jira/WEB-INF/classes/lib/

While under home directory - you will have it under ;

.../Application Data/JIRA/plugins/installed-plugins

[Jason Huntowski]

Vickey,

Thanks for the info. I see now where these files are located. This definitely helps me to learn.

Thanks,

Jason