Failed to validate connection com.mysql.jdbc.JDBC4Connection

[stevelmc]

Hello,

I'm running Bitbucket Server 6.7.1 on RHEL7. It's using a MariaDB 5.5.64 database. When the server restarts, Bitbucket is unable to restore connection to the database pool. I get these messages in atlassian-bitbucket.log and the service will not start:

2019-11-22 06:11:28,981 INFO [Thread-5] c.a.b.i.boot.log.BuildInfoLogger Bitbucket 6.7.1 has shut down
2019-11-22 06:11:33,147 WARN [Thread-5] com.zaxxer.hikari.pool.PoolBase bitbucket - Failed to validate connection com.mysql.jdbc.JDBC4Connection@740e4c8b (No operations allowed after
connection closed.)
2019-11-22 06:11:33,150 WARN [Thread-5] com.zaxxer.hikari.pool.PoolBase bitbucket - Failed to validate connection com.mysql.jdbc.JDBC4Connection@2d3db937 (No operations allowed after
connection closed.)
2019-11-22 06:11:33,152 WARN [Thread-5] com.zaxxer.hikari.pool.PoolBase bitbucket - Failed to validate connection com.mysql.jdbc.JDBC4Connection@59b587ad (No operations allowed after connection closed.)
2019-11-22 06:11:33,157 WARN [Thread-5] com.zaxxer.hikari.pool.PoolBase bitbucket - Failed to validate connection com.mysql.jdbc.JDBC4Connection@5f59e204 (No operations allowed after connection closed.)
2019-11-22 06:11:33,161 WARN [Thread-5] com.zaxxer.hikari.pool.PoolBase bitbucket - Failed to validate connection com.mysql.jdbc.JDBC4Connection@25b52d4c (No operations allowed after connection closed.)
2019-11-22 06:11:48,091 WARN [Thread-5] o.h.e.jdbc.spi.SqlExceptionHelper SQL Error: 0, SQLState: 08003
2019-11-22 06:11:48,093 ERROR [Thread-5] o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15001ms.
2019-11-22 06:11:48,093 ERROR [Thread-5] o.h.e.jdbc.spi.SqlExceptionHelper No operations allowed after connection closed.
2019-11-22 06:11:48,132 ERROR [Thread-5] c.a.e.i.AsynchronousAbleEventDispatcher There was an exception thrown trying to dispatch event [com.atlassian.config.lifecycle.events.ApplicationStoppingEvent@42f47703] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.analytics.client.listener.BitbucketEventListener.onEvent(java.lang.Object), listener=com.atlassian.analytics.client.listener.BitbucketEventListener@4780c70d} (monitored)]
java.lang.RuntimeException: could not prepare statement. Listener: com.atlassian.analytics.client.listener.BitbucketEventListener event: com.atlassian.config.lifecycle.events.ApplicationStoppingEvent
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:57)
at com.atlassian.diagnostics.internal.platform.event.EventSystemMonitor.invokeMonitored(EventSystemMonitor.java:109)
at com.atlassian.diagnostics.internal.platform.event.MonitoredListenerInvoker.invoke(MonitoredListenerInvoker.java:38)
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.lambda$null$0(AsynchronousAbleEventDispatcher.java:37)
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:85)
at com.atlassian.diagnostics.internal.platform.event.MonitoredEventDispatcher.dispatch(MonitoredEventDispatcher.java:36)
at com.atlassian.event.internal.LockFreeEventPublisher$Publisher.dispatch(LockFreeEventPublisher.java:215)
at com.atlassian.event.internal.LockFreeEventPublisher.publish(LockFreeEventPublisher.java:88)
at com.atlassian.stash.internal.event.TransactionAwareEventPublisher.internalPublish(TransactionAwareEventPublisher.java:155)
at com.atlassian.stash.internal.event.TransactionAwareEventPublisher.publish(TransactionAwareEventPublisher.java:99)
at com.atlassian.stash.internal.config.ConfigLifecycle.stop(ConfigLifecycle.java:45)
at com.atlassian.stash.internal.spring.AbstractSmartLifecycle.stop(AbstractSmartLifecycle.java:36)
at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:238)
at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:53)

 The database is up and I can access it via the command line.

Does anyone have any ideas as to how to get the service back up and running?

[stevelmc]

This ended up being caused by the JRE_HOME setting in set-jre-home.sh. It had previously been set in Stash's setenv.sh in JAVA_HOME to a symlinked path. When upgrading to Bitbucket, it expanded the path to include the current version of OpenJDK at the time. When the OpenJDK package was updated on the server, that path was no longer valid so the application would not start. Fixing the JRE_HOME corrected the issue.