Bitbucket ElasticSearch Service will not start after Upgrade

[Maggie]

I upgraded to Bitbucket Server 7.0 yesterday. Atlassian Bitbucket service starts, but Atlassian Bitbucket Elasticsearch service does not. I am receiving the following error:

Windows Event Viewer:

The Atlassian Bitbucket Elasticsearch 7.5.2 (AtlassianBitbucketElasticsearch) service terminated with the following service-specific error: 
The system cannot open the file.

System Info:

• Microsoft Windows Server 2019, 64-bit
• MS SQL Server 2017
• Java JDK 11

I've checked the following:

• JAVA_HOME & JRE_HOME environment variables:  It's been unclear what JRE_HOME should be when using JDK 11 (since Java no longer bundles the JRE separately). I've tried setting JRE_HOME to be equal to JAVA_HOME, as well as setting it to [JAVA_HOME]\bin.
• Services: I used Bitbucket & Bitbucket ElasticSearch Configuration Managers to update the paths on the Java tabs of the services.
• IP address: I checked the bitbucket.properties file for any references to IP address. There were none.

Other Circumstances:

• Hosted on a virtual server: Server is a virtual machine, which was moved earlier this week. Server name did not change, but the IP address did.
• Indexes: When reinstalling, the Bitbucket installation location was moved. I found the help article below suggesting that I reindex. The <BITBUCKET_HOME>/shared/data folder did not contain a nodes subfolder, let alone any indexes within it. I tried following the suggestions in the article below. I could not get the first suggestion to work, & there was no nodes folder for the indexes to repopulate into RE: the second suggestion.
  
  https://confluence.atlassian.com/bitbucketserverkb/bitbucket-server-windows-service-fails-to-start-the-system-cannot-open-the-file-779171450.html

Here is a sample of my atlassian-bitbucket.log file:

2020-05-29 11:49:08,961 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-05-29 11:49:08,961 WARN [Caesium-1-4] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: c94b9608-104c-49aa-bbfd-73e045895b8e
2020-05-29 11:49:08,961 DEBUG [Caesium-1-4] c.a.b.i.s.c.c.ClusterJobScheduler Rescheduling failed job. Current retry: 1
2020-05-29 11:49:12,948 DEBUG [Caesium-1-2] c.a.b.i.s.c.j.c.HealthCheckRunner Running up to 4 search health checks
2020-05-29 11:49:12,948 DEBUG [Caesium-1-2] c.a.b.i.s.c.c.DefaultElasticsearchClientFactory Create new inspecting Elasticsearch client using server URL 'http://fsstllabvm2:7992/'
2020-05-29 11:49:13,995 DEBUG [pool-17-thread-1] c.a.b.i.s.c.s.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed due to exception: 
java.util.concurrent.CompletionException: java.net.ConnectException: Connection refused: no further information
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)...

2020-05-29 11:49:15,011 DEBUG [Caesium-1-2] c.a.b.i.s.i.j.checks.KnownIssueCheck Index previously repaired on Tue Jul 30 10:59:07 CDT 2019. Skipping.
2020-05-29 11:49:15,011 DEBUG [Caesium-1-2] c.a.b.i.s.c.j.c.HealthCheckRunner Recovery result for KnownIssueCheck was RESOLVED
2020-05-29 11:49:15,011 DEBUG [Caesium-1-2] c.a.b.i.s.c.j.c.HealthCheckRunner Health checks will run again after PT5M
2020-05-29 11:50:08,970 DEBUG [Caesium-1-1] c.a.b.i.s.c.cluster.ClusterJobRunner Job has started: 7f214192-82e9-4bac-bf9b-4fc22502d24f
2020-05-29 11:50:08,970 DEBUG [Caesium-1-1] c.a.b.i.s.i.jobs.StartupChecksJob Attempting provisioning in startup job
2020-05-29 11:50:08,970 DEBUG [Caesium-1-1] c.a.b.i.s.c.s.p.AutomaticAuthenticationProvisioner Testing if automatic Elasticsearch authentication configuration should be performed.
2020-05-29 11:50:08,970 DEBUG [Caesium-1-1] c.a.b.i.s.c.s.p.AutomaticAuthenticationProvisioner Skipping automatic auth configuration: Username or password have already been set.
2020-05-29 11:50:08,970 INFO [Caesium-1-1] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2020-05-29 11:50:09,987 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-05-29 11:50:09,987 WARN [Caesium-1-1] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: 7f214192-82e9-4bac-bf9b-4fc22502d24f
2020-05-29 11:50:09,987 DEBUG [Caesium-1-1] c.a.b.i.s.c.c.ClusterJobScheduler Rescheduling failed job. Current retry: 2
2020-05-29 11:50:25,955 INFO [Thread-61] c.a.b.i.boot.log.BuildInfoLogger Bitbucket 7.2.3 has shut down
2020-05-29 11:50:26,063 DEBUG [Thread-61] c.a.s.i.c.HazelcastBucketedExecutor Executor service has shutdown gracefully
2020-05-29 11:50:26,063 DEBUG [Thread-61] c.a.s.i.s.ScheduledJobLifecycle Unscheduling jobs for RateLimitReaperJob
2020-05-29 11:50:26,063 DEBUG [Thread-61] c.a.s.i.s.ScheduledJobLifecycle Unscheduling jobs for HistoryFlushJob
2020-05-29 11:50:26,095 DEBUG [Thread-61] c.a.s.i.c.g.c.CachingCommitGraphSource Executor service has shutdown gracefully
2020-05-29 11:50:26,095 INFO [Thread-61] c.a.p.manager.DefaultPluginManager Preparing to shut down the plugin system

2020-05-29 11:50:26,095 DEBUG [Thread-61] c.a.b.i.s.i.e.DefaultIndexEventQueueProcessor Event processor is now entering STOPPED state
2020-05-29 11:50:28,111 INFO [Thread-61] c.a.b.i.s.i.e.DefaultIndexEventQueueProcessor Worker thread has been shut down
2020-05-29 11:50:28,111 DEBUG [Thread-61] c.a.s.s.a.l.PatchedLifecycleManager Calling LifecycleAware.onStop() 'com.atlassian.bitbucket.internal.plugin.extender.BeanEventListenerScanner$EventListenerLifecycle@332692bc' from plugin 'com.atlassian.bitbucket.server.bitbucket-default-reviewers'

2020-05-29 11:50:28,111 INFO [Thread-61] c.a.b.internal.ssh.server.SshServer Stopping SSH server...
2020-05-29 11:50:28,126 INFO [Thread-61] c.a.b.internal.ssh.server.SshServer Stopped SSH server successfully.
2020-05-29 11:50:28,126 DEBUG [Thread-61] c.a.s.s.a.l.PatchedLifecycleManager Calling LifecycleAware.onStop() 'com.atlassian.bitbucket.internal.plugin.extender.BeanEventListenerScanner$EventListenerLifecycle@d36562f' from plugin 'com.atlassian.plugins.authentication.atlassian-authentication-plugin'

2020-05-29 11:50:28,157 INFO [Thread-61] c.a.p.manager.DefaultPluginManager Shutting down the plugin system
2020-05-29 11:50:28,220 INFO [Gemini Blueprint context shutdown thread 1] o.s.o.a.NonValidatingOsgiBundleXmlApplicationContext Unpublishing application context OSGi service for bundle Bitbucket Japanese (Japan) Language Pack (tac.bitbucket.languages.ja_JP)

2020-05-29 11:50:29,282 INFO [FelixStartLevel] c.a.p.osgi.factory.OsgiBundlePlugin Plugin 'rome.rome-1.0' bundle stopped: ENABLED
2020-05-29 11:50:29,282 WARN [FelixStartLevel] c.a.p.osgi.factory.OsgiBundlePlugin Cannot disable Bundle 'rome.rome-1.0', not ACTIVE

2020-05-29 11:50:29,314 INFO [FelixStartLevel] c.a.p.o.f.ModuleDescriptorServiceTrackerCustomizer Dynamically removed module descriptor: com.atlassian.bitbucket.server.bitbucket-bundled-hooks:requiredApproversMergeHook-config-form

2020-05-29 11:50:29,407 WARN [Thread-61] c.a.p.osgi.factory.OsgiBundlePlugin OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2020-05-29 11:50:29,486 DEBUG [Thread-61] c.a.s.i.c.HazelcastBucketedExecutor Executor service has shutdown gracefully
2020-05-29 11:50:29,501 DEBUG [Thread-61] c.a.s.i.s.g.t.DefaultTranscodeServer Closing callback socket
2020-05-29 11:50:29,501 DEBUG [git-transcode-listener] c.a.s.i.s.g.t.DefaultTranscodeServer The server socket has been closed; stopping...
2020-05-29 11:50:29,501 DEBUG [Thread-61] c.a.s.i.s.g.t.DefaultTranscodeServer The acceptor thread has stopped cleanly
2020-05-29 11:50:29,517 INFO [Thread-61] o.s.s.c.ThreadPoolExecutorFactoryBean Shutting down ExecutorService 'passwordResetExecutorService'
2020-05-29 11:50:29,532 DEBUG [Thread-61] c.a.s.i.hook.HookThreadPoolExecutor Executor service has shutdown gracefully
2020-05-29 11:50:29,532 DEBUG [Thread-61] c.a.b.i.p.nu.NuNioProcessHelper Executor service has shutdown gracefully

2020-05-29 11:50:29,548 DEBUG [Thread-61] c.a.s.i.c.HazelcastBucketedExecutor Executor service has shutdown gracefully

2020-05-29 11:50:29,548 DEBUG [Thread-61] c.a.s.i.i.DefaultCommitIndexingService IndexingService is shutting down. Running indexing jobs will be aborted
2020-05-29 11:50:29,579 INFO [Thread-61] o.s.s.c.ThreadPoolExecutorFactoryBean Shutting down ExecutorService 'org.springframework.scheduling.concurrent.ThreadPoolExecutorFactoryBean#2e28c858'
2020-05-29 11:50:29,595 DEBUG [Thread-61] c.a.s.i.topic.HazelcastTopicService Executor service has shutdown gracefully
2020-05-29 11:50:29,611 INFO [Thread-61] o.h.internal.SessionFactoryRegistry HHH000097: Unbound factory from JNDI name: bitbucket.core
2020-05-29 11:50:29,611 INFO [Thread-61] c.a.s.internal.home.HomeLockAcquirer Releasing lock on C:\Program Files\Atlassian\Application Data\Bitbucket\shared
2020-05-29 11:50:29,611 DEBUG [Thread-61] c.a.s.i.e.SingletonEventExecutorFactory Executor service has shutdown gracefully
2020-05-29 11:50:29,611 INFO [Thread-61] com.zaxxer.hikari.HikariDataSource bitbucket - Shutdown initiated...
2020-05-29 11:50:29,611 INFO [Thread-61] com.zaxxer.hikari.HikariDataSource bitbucket - Shutdown completed.
2020-05-29 11:50:29,611 INFO [Thread-61] c.a.s.internal.home.HomeLockAcquirer Releasing lock on C:\Program Files\Atlassian\Application Data\Bitbucket
2020-05-29 11:50:29,626 DEBUG [Thread-61] c.a.s.i.w.c.ContextCleanupListener Shutting down Hazelcast instance: hazelcast
2020-05-29 11:50:29,626 INFO [Thread-61] com.hazelcast.core.LifecycleService [10.0.74.44]:5701 [atlbitbucket] [3.12.6] [10.0.74.44]:5701 is SHUTTING_DOWN
2020-05-29 11:50:29,626 INFO [hz.hazelcast.cached.thread-6] c.h.i.p.impl.MigrationManager [10.0.74.44]:5701 [atlbitbucket] [3.12.6] Shutdown request of Member [10.0.74.44]:5701 - 23df2613-9656-419d-9b9a-d8524da1fd70 this is handled
2020-05-29 11:50:29,642 INFO [Thread-61] com.hazelcast.instance.Node [10.0.74.44]:5701 [atlbitbucket] [3.12.6] Shutting down connection manager...
2020-05-29 11:50:29,642 INFO [Thread-61] com.hazelcast.instance.Node [10.0.74.44]:5701 [atlbitbucket] [3.12.6] Shutting down node engine...
2020-05-29 11:50:29,673 INFO [Thread-61] com.hazelcast.instance.NodeExtension [10.0.74.44]:5701 [atlbitbucket] [3.12.6] Destroying node NodeExtension.
2020-05-29 11:50:29,673 INFO [Thread-61] com.hazelcast.instance.Node [10.0.74.44]:5701 [atlbitbucket] [3.12.6] Hazelcast Shutdown is completed in 47 ms.
2020-05-29 11:50:29,673 INFO [Thread-61] com.hazelcast.core.LifecycleService [10.0.74.44]:5701 [atlbitbucket] [3.12.6] [10.0.74.44]:5701 is SHUTDOWN
2020-05-29 11:50:29,673 DEBUG [Thread-61] c.a.s.i.w.c.ContextCleanupListener Unregistering JDBC driver: com.microsoft.sqlserver.jdbc.SQLServerDriver
2020-05-29 11:50:29,673 DEBUG [Thread-61] c.a.s.i.w.c.ContextCleanupListener Unregistering JDBC driver: org.h2.Driver
2020-05-29 11:50:29,673 DEBUG [Thread-61] c.a.s.i.w.c.ContextCleanupListener Unregistering JDBC driver: org.hsqldb.jdbc.JDBCDriver

One of my teammates is receiving the following error when trying to do a PUSH:

Error encountered when pushing to the remote repository: rejected master -> master (pre-received hook declined)

I am guessing that this is a result of the above-mentioned issues.

Any assistance would be most appreciated!

[Maggie]

I tried uninstalling & reinstalling the Bitbucket Elasticsearch service. Instead of the error mentioned above, I'm now getting the following error:

The Atlassian Bitbucket Elasticsearch service terminated unexpectedly.

Reinstalling was very difficult because the elasticsearch-service.bat file referenced a directory that had a slightly different name (config-template instead of config). I had to make a copy & rename. Once I was able to run the bat file, the service it created had the wrong name. I had to modified the script, & then further customize the service once it was created.