JIRA does not start after reboot

[neic]

Hey. I have Bitbucket and JIRA on the same server.

After rebooting Bitbucket starts up normally, JIRA is not.

After reboot in JIRA log not exists any logs for startup after shutdown logs.

If I start JIRA a little later manually, it starts without errors.

JIRA log (after shutdown):

...

2019-03-03 11:59:21,674 Gemini Blueprint context shutdown thread 1 WARN [o.s.context.annotation.CommonAnnotationBeanPostProcessor] Invocation of destroy method failed on bean with name 'refreshConnecti$
2019-03-03 11:59:22,200 Gemini Blueprint context shutdown thread 1 WARN [o.o.c.entity.jdbc.SQLProcessor] [SQLProcessor.rollback]: SQL Exception while rolling back insert. Error was:org.postgresql.util.$
2019-03-03 11:59:22,200 Gemini Blueprint context shutdown thread 1 WARN [NoModule] org.ofbiz.core.entity.jdbc.SQLProcessor
org.postgresql.util.PSQLException: This connection has been closed.
at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:803)
at org.postgresql.jdbc.PgConnection.rollback(PgConnection.java:810)
at org.apache.commons.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:488)
at org.apache.commons.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:488)
at org.ofbiz.core.entity.jdbc.interceptors.connection.DelegatingConnection.rollback(DelegatingConnection.java:61)
at org.ofbiz.core.entity.jdbc.SQLProcessor.rollback(SQLProcessor.java:214)
at org.ofbiz.core.entity.GenericDAO.deleteByAnd(GenericDAO.java:1250)
at org.ofbiz.core.entity.GenericHelperDAO.removeByAnd(GenericHelperDAO.java:236)
at org.ofbiz.core.entity.GenericDelegator.removeByAnd(GenericDelegator.java:1334)
at org.ofbiz.core.entity.GenericDelegator.removeByAnd(GenericDelegator.java:1313)
at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.removeByAnd(DefaultOfBizDelegator.java:236)
at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.removeByAnd(WrappingOfBizDelegator.java:131)
at com.atlassian.jira.entity.Delete$DeleteWhereContext.execute(Delete.java:141)
at com.atlassian.jira.entity.EntityEngineImpl.delete(EntityEngineImpl.java:55)
at com.atlassian.jira.entity.Delete$DeleteWhereContext.execute(Delete.java:136)
at com.atlassian.jira.scheduler.OfBizClusteredJobDao.delete(OfBizClusteredJobDao.java:152)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.unscheduleJob(CaesiumSchedulerService.java:208)
at com.atlassian.scheduler.core.DelegatingSchedulerService.unscheduleJob(DelegatingSchedulerService.java:73)
... 2 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
at com.sun.proxy.$Proxy574.unscheduleJob(Unknown Source)
... 2 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy712.unscheduleJob(Unknown Source)
at com.atlassian.jira.apdex.impl.SendAnalyticsJobRunner.destroy(SendAnalyticsJobRunner.java:91)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:256)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:571)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:543)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:954)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:504)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:961)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1039)
at org.eclipse.gemini.blueprint.context.support.AbstractOsgiBundleApplicationContext.destroyBeans(AbstractOsgiBundleApplicationContext.java:194)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1015)
at org.eclipse.gemini.blueprint.context.support.AbstractOsgiBundleApplicationContext.doClose(AbstractOsgiBundleApplicationContext.java:187)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$501(AbstractDelegatedExecutionApplicationContext.java:57)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$2.run(AbstractDelegatedExecutionApplicationContext.java:191)
at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.normalClose(AbstractDelegatedExecutionApplicationContext.java:187)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.close(DependencyWaiterApplicationContextExecutor.java:382)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.doClose(AbstractDelegatedExecutionApplicationContext.java:211)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:965)
at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager.closeApplicationContext(LifecycleManager.java:332)
at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager.access$200(LifecycleManager.java:51)
at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager$2.run(LifecycleManager.java:297)
at org.eclipse.gemini.blueprint.extender.internal.util.concurrent.RunnableTimedExecution$MonitoredRunnable.run(RunnableTimedExecution.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-03-03 11:59:22,202 Gemini Blueprint context shutdown thread 1 WARN [o.s.b.factory.support.DisposableBeanAdapter] Invocation of destroy method failed on bean with name 'sendAnalyticsJobRunner': com$
2019-03-03 11:59:24,512 Gemini Blueprint context shutdown thread 1 INFO [c.a.j.plugins.monitor.MonitoringScheduler] Unscheduling metrics collector...
2019-03-03 11:59:24,512 Gemini Blueprint context shutdown thread 1 INFO [c.a.j.plugins.monitor.MonitorLauncher] Stopped JIRA monitoring
2019-03-03 11:59:26,295 localhost-startStop-2 INFO [c.a.jira.startup.DefaultJiraLauncher] JIRA launchers stopped in 14208ms

[Andy Heinzer]

What version of Jira is this?  I ask because there was a known bug that could cause Jira to fail to start up, IF an end user requested any jsp page from Jira during the initial startup.  This is documented in JRASERVER-62486  This problem affected a lot of different Jira versions.

I cannot say for you are seeing this specific problem, but this is one way I know that Jira startup can fail in some recent versions.  

If you are already running a version of Jira that contains this fix, then we can safely rule out this cause.  In that case, please tell us more about your environment:

1. What operating system is this?
2. How are you starting Jira and bitbucket? (via a service, or script, etc)
3. I would also be interested to see more of your logs, I can see there was a SQL exception, but I'm interested to learn more about the specific details of that exception.  It might be in a previous stacktrace in the logs.

[neic]

Hi, thanks for answer. I use:

Jira Software 8.0.0#800007.

Ubuntu 18.04.2 LTS

Jira and Bitbucket start as a service.

I took the following steps:

1) Clear the log file

2) Reboot

3) After rebooting, jira does not start automatically and I started it manually

4) I'm waiting for jira to start completely (I don't go to jira)

5) Reboot

6) Jira does not start automatically

Last errors and until to end file:

https://yadi.sk/i/KI3VLJdCPZUzYg

[neic]

Sorry for link

[Andy Heinzer]

So far I haven't actually seen any logs of Jira starting up.    You can also look in the $JIRAINSTALL/log/catalina.out when Jira is trying to start up, in case the atlassian-jira.log file is not yet being updated by a failed startup attempt.   That would better help us to understand what Jira is doing when trying to startup if it fails.

From looking at the logs you have posted so far, I see this error being thrown:

Database:
- name:PostgreSQL
- version:10.7 (Ubuntu 10.7-1.pgdg18.04+1)
- minor version:7
- major version:10
Driver:
- name:PostgreSQL Native Driver
- version:PostgreSQL 9.4.1212
org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command

These logs are when Jira is already in the process of shutting down, or the operating system itself is.  This terminating connection due to administrator command is expected when the operating system admin has issued a reboot of the entire operating system.  What I suspect has happened is that the Postgresql service has stopped, but Jira is technically still running at the time.  It would be a very small amount of time I imagine (a few seconds to maybe as long as a minute or two), but this kind of shutdown could be problematic for Jira.  Since Jira is expecting that when it is running that SQL will also be up, running, and available, to have the system shut down SQL first seems to take the legs out from under Jira. 

This is bad because Jira isn't being given the opportunity to shutdown gracefully and save any cached information not yet stored to the database.  Ideally, if you know you are going to be rebooting the system, I would recommend that you stop Jira first before rebooting the OS.  This at least gives the application to safely close the connections it has open to the SQL database.  

But there is another problem revealed with your error message.   And that is the version of Postgresql you are running here is not yet supported by Jira.  Jira 8 does support postgresql, but currently that support is limited to versions 9.4-9.6.x

This database version issue might not be the cause of the problem you are seeing here, but since it's an unsupported platform, I would still recommend that you migrate this data to a supported database to make sure this isn't a contributing factor to the performance issue here.   Steps to do this are in Switching databases.

[neic]

I took the following steps:

1) Start JIRA manually after reboot

2) Waiting for jira to start completely

3) Stop JIRA

4) Reboot

Actually: JIRA didn't start.

I think this problem possible associate with bugs on install. If not, then with DB.

I will try reset my server and install JIRA on clear server again (with PostgresSQL 10.7, and last support DB version if I will have problems with auto start).

I will write message here about results.

[neic]

Hi, i did try install JIRA with PostgreSQL 10 and PostgresSQL 9.6, and without DB (just install only JIRA and after install -reboot).

Result is one - JIRA not start automatically =(

May be problems in OS?

[neic]

Heh, yes in Ubuntu 16.04 all is ok.