Starting cluster Node1 immediately shuts down after the startup

[Dong Park]

We have been operating a single, independent server under a datacenter license. Our goal is now to set it up as a cluster, utilizing two nodes. However, we are encountering an issue where Jira initiates and then promptly shuts down. The log files don't provide much clarity on this matter.

Here is what we did.

Under Windows Server 2016 Standard & Jira Software 8.20.19. 

Copied the directories to  \\jira-data\data\jira-shared

• data
• plugins
• logos
• import
• export
• caches
• keys

cluster.properties

jira.node.id = node1
jira.shared.home = \\\\jira-data\\data\\jira-shared

atlassian-jira.log

2024-01-05 21:18:38,191-0500 localhost-startStop-1 INFO [c.a.jira.startup.LauncherContextListener] Startup is complete. Jira is ready to serve.
2024-01-05 21:18:38,206-0500 localhost-startStop-1 INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 2757 MiB. Committed: 4096 MiB. Max: 12000 MiB
Non-heap memory : Used: 455 MiB. Committed: 494 MiB. Max: 1536 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 3212 MiB. Committed: 4590 MiB. Max: 13536 MiB
---------------------------------------------------------------------------------
2024-01-05 21:18:38,505-0500 localhost-startStop-1 INFO [c.a.jira.startup.DefaultJiraLauncher] Stopping launchers
2024-01-05 21:18:38,505-0500 localhost-startStop-1 INFO [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Pausing node re-index service
java.lang.Exception
at com.atlassian.jira.index.ha.DefaultNodeReindexService.pause(DefaultNodeReindexService.java:216)
at com.atlassian.jira.index.ha.DefaultNodeReindexService.shutDown(DefaultNodeReindexService.java:185)
at com.atlassian.jira.cluster.DefaultClusterServicesManager.stopServices(DefaultClusterServicesManager.java:95)
at com.atlassian.jira.startup.ClusteringLauncher.stop(ClusteringLauncher.java:51)
at com.atlassian.jira.startup.DefaultJiraLauncher.stop(DefaultJiraLauncher.java:198)
at com.atlassian.jira.startup.LauncherContextListener.contextDestroyed(LauncherContextListener.java:212)
... 5 filtered
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2024-01-05 21:18:38,505-0500 localhost-startStop-1 INFO [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Shutting down replication thread
java.lang.Exception
at com.atlassian.jira.index.ha.DefaultNodeReindexService.shutDown(DefaultNodeReindexService.java:186)
at com.atlassian.jira.cluster.DefaultClusterServicesManager.stopServices(DefaultClusterServicesManager.java:95)
at com.atlassian.jira.startup.ClusteringLauncher.stop(ClusteringLauncher.java:51)
at com.atlassian.jira.startup.DefaultJiraLauncher.stop(DefaultJiraLauncher.java:198)
at com.atlassian.jira.startup.LauncherContextListener.contextDestroyed(LauncherContextListener.java:212)
... 5 filtered
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2024-01-05 21:18:38,505-0500 localhost-startStop-1 INFO

jirasoftware stderr

05-Jan-2024 21:16:15.573 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
05-Jan-2024 21:16:15.573 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/8.5.85]
Security framework of XStream not explicitly initialized, using predefined black list on your own risk.
Security framework of XStream not explicitly initialized, using predefined black list on your own risk.
05-Jan-2024 21:18:38.456 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-openssl-nio-443"]
05-Jan-2024 21:18:38.472 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 142936 ms
05-Jan-2024 21:18:38.472 SEVERE [main] org.apache.catalina.core.StandardServer.await Failed to create server shutdown socket on address [localhost] and port [8005] (base port [8005] and offset [0])
java.net.BindException: Address already in use: NET_Bind
at java.base/java.net.PlainSocketImpl.bind0(Native Method)
at java.base/java.net.PlainSocketImpl.socketBind(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.bind(Unknown Source)
at java.base/java.net.ServerSocket.bind(Unknown Source)
at java.base/java.net.ServerSocket.<init>(Unknown Source)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:446)
at org.apache.catalina.startup.Catalina.await(Catalina.java:783)
at org.apache.catalina.startup.Catalina.start(Catalina.java:729)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:347)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:478)
05-Jan-2024 21:18:38.488 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-openssl-nio-443"]
05-Jan-2024 21:18:38.499 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
05-Jan-2024 21:19:08.553 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [jira-stats-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

jirasoftware stdout

2024-01-05 21:18:38,191-0500 localhost-startStop-1 INFO [c.a.jira.startup.LauncherContextListener] Startup is complete. Jira is ready to serve.
2024-01-05 21:18:38,206-0500 localhost-startStop-1 INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 2757 MiB. Committed: 4096 MiB. Max: 12000 MiB
Non-heap memory : Used: 455 MiB. Committed: 494 MiB. Max: 1536 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 3212 MiB. Committed: 4590 MiB. Max: 13536 MiB
---------------------------------------------------------------------------------
2024-01-05 21:18:38,505-0500 localhost-startStop-1 INFO [c.a.jira.startup.DefaultJiraLauncher] Stopping launchers
2024-01-05 21:18:38,802-0500 Caesium-1-2 INFO ServiceRunner [c.a.r.internal.configuration.ConfigurationLoggerJob] Periodic rate limiting configuration log. System rate limiting settings: [SystemRateLimitingSettings(mode=OFF, bucketSettings=TokenBucketSettings(capacity=50, fillRate=10, intervalFrequency=1, intervalTimeUnit=Seconds), jobControlSettings=SystemJobControlSettings(reportingDbArchivingJobFrequencyDuration=PT1M10S, reportingDbRetentionPeriodDuration=PT24H, bucketCollectionJobFrequencyDuration=PT5M, bucketCleanupJobFrequencyDuration=PT15M, settingsReloadJobFrequencyDuration=PT1M))]
2024-01-05 21:19:08,521-0500 localhost-startStop-1 INFO [c.a.jira.startup.DefaultJiraLauncher] JIRA launchers stopped in 30016ms

catalian log

05-Jan-2024 21:16:15.291 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-nio-443"]
05-Jan-2024 21:16:15.541 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 983 ms
05-Jan-2024 21:16:15.573 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
05-Jan-2024 21:16:15.573 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/8.5.85]
05-Jan-2024 21:18:38.456 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-openssl-nio-443"]
05-Jan-2024 21:18:38.472 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 142936 ms
05-Jan-2024 21:18:38.472 SEVERE [main] org.apache.catalina.core.StandardServer.await Failed to create server shutdown socket on address [localhost] and port [8005] (base port [8005] and offset [0])
java.net.BindException: Address already in use: NET_Bind
at java.base/java.net.PlainSocketImpl.bind0(Native Method)
at java.base/java.net.PlainSocketImpl.socketBind(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.bind(Unknown Source)
at java.base/java.net.ServerSocket.bind(Unknown Source)
at java.base/java.net.ServerSocket.<init>(Unknown Source)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:446)
at org.apache.catalina.startup.Catalina.await(Catalina.java:783)
at org.apache.catalina.startup.Catalina.start(Catalina.java:729)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:347)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:478)
05-Jan-2024 21:18:38.488 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-openssl-nio-443"]
05-Jan-2024 21:18:38.499 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]

[Nic Brough -Adaptavist-]

Hi,

The error message is clear - 

Failed to create server shutdown socket on address [localhost] and port [8005] (base port [8005] and offset [0])
java.net.BindException: Address already in use: NET_Bind

Something on that machine is already running a service on port 8005.

You need to identify what it is (it may be that your Jira service is already running!) and either stop and disable it, or configure it or your new service to run on different ports.

[Dong Park]

I did observe that error, but initially, I believed it was an unrelated issue when attempting to shut down. However, after removing the cluster.properties file, the Jira server successfully starts.

In cluster mode, the Windows service remains in the running state, yet it continuously attempts to shut down the process.  I do think the port 8005 is something I would need to look into but I doubt it's related to cluster.  

Thanks for your inputs though. 

[Nic Brough -Adaptavist-]

It is the same answer - something is already using that port

Possibly not Jira, but the simplest thing to do is look at what is using your ports.  On a good OS, ss, or netstat -plant run on the command line will tell you.