how to add a node in cluster (kubernetes env)

[Alexander Tchaikovski]

Hello Community! Could someone please point me to the right direction on how to add a node to confluence in kubernetes. I've installed it in mono by helm chart with values.yml tuned to my environment. local-home is on pv/pvc (readwriteonce), shared home is on readwritemany. database is postgres in clustering mode, works ok. confluence starts ok without errors then i configure it then start 2nd pod by scaling stateful set. 2nd pod starts ok but how to point it to the first one? tried replacing config the same as 1st one (with and without changing server.id) - won't work. is there a document or git repo with a correct one? please. :) running out of time and ideas. thank you!!

[Yevhen · Accepted solution]

@Alexander Tchaikovski can you please elaborate on "how to point it to the first one"? Confluence uses Hazelcast K8s discovery and when you scale up and the 2nd pod starts without errors and passes a health check, things should happen automatically for you and a new member should join the Hazelcast cluster.

Sharing logs of both pods will help too.

[Alexander Tchaikovski]

@Yevhen hello. my experience is too short to discuss it deeper. just walking in the darkness. here're 2 logs downloaded from k8s. not found attach file here so will past here: 

pod-0.log:

26-Apr-2022 16:42:45.212 INFO [http-nio-8090-exec-39] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:42:45.716 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:42:45.948 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc

26-Apr-2022 16:42:45.952 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor

26-Apr-2022 16:42:45.952 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport

26-Apr-2022 16:42:45.974 INFO [http-nio-8090-exec-29] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:42:47.564 INFO [http-nio-8090-exec-14] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:42:47.661 INFO [http-nio-8090-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:42:47.670 INFO [http-nio-8090-exec-14] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc

26-Apr-2022 16:42:47.670 INFO [http-nio-8090-exec-14] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor

26-Apr-2022 16:42:47.671 INFO [http-nio-8090-exec-14] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport

26-Apr-2022 16:42:47.822 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:42:47.983 INFO [http-nio-8090-exec-8] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc

26-Apr-2022 16:42:47.984 INFO [http-nio-8090-exec-8] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor

26-Apr-2022 16:42:47.984 INFO [http-nio-8090-exec-8] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport

26-Apr-2022 16:42:47.990 INFO [http-nio-8090-exec-10] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:42:48.145 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc

26-Apr-2022 16:42:48.145 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor

26-Apr-2022 16:42:48.145 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport

26-Apr-2022 16:42:48.387 INFO [http-nio-8090-exec-37] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:43:09.172 INFO [http-nio-8090-exec-9] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:43:09.346 WARNING [http-nio-8090-exec-9] com.sun.jersey.spi.inject.Errors.processErrorMessages The following warnings have been detected with resource and/or provider classes:

WARNING: A HTTP GET method, public javax.ws.rs.core.Response com.atlassian.confluence.tinymceplugin.rest.DraftsResource.getDrafts(int,int), should not consume any entity.

26-Apr-2022 16:43:19.576 INFO [http-nio-8090-exec-10] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:43:36.868 INFO [http-nio-8090-exec-36] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:43:49.577 INFO [http-nio-8090-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:51:17.085 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:51:22.197 INFO [http-nio-8090-exec-29] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:51:40.631 INFO [http-nio-8090-exec-36] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

26-Apr-2022 16:51:42.497 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

27-Apr-2022 03:34:32.372 INFO [http-nio-8090-exec-37] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

27-Apr-2022 03:34:32.391 INFO [http-nio-8090-exec-23] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

27-Apr-2022 03:34:32.453 INFO [http-nio-8090-exec-7] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

27-Apr-2022 03:34:32.701 WARNING [http-nio-8090-exec-7] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://localhost:54350/rest/feature-discovery/1.0/discovered/com.atlassian.confluence.plugins.confluence-dashboard/transition-saved-for-later, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

27-Apr-2022 03:37:13.566 WARNING [http-nio-8090-exec-10] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://wiki.yc.hh/rest/feature-discovery/1.0/discovered/com.atlassian.confluence.plugins.confluence-dashboard/transition-saved-for-later, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expect

pod-1.log:

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out

---------------------------------------------------------------------------

Using Java: /opt/java/openjdk/bin/java

2022-04-26 16:47:07,996 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required

---------------------------------------------------------------------------

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

[0.000s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2022-04-26_16-47-08.log instead.

26-Apr-2022 16:47:08.333 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]

26-Apr-2022 16:47:08.345 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [proxyPort] to []

26-Apr-2022 16:47:08.355 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine] failed to set property [debug] to [0]

26-Apr-2022 16:47:08.360 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [debug] to [0]

26-Apr-2022 16:47:08.389 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]

26-Apr-2022 16:47:08.409 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context/Valve] failed to set property [resolveHosts] to [false]

26-Apr-2022 16:47:08.410 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]

26-Apr-2022 16:47:08.626 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]

26-Apr-2022 16:47:08.645 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [424] milliseconds

26-Apr-2022 16:47:08.651 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]

26-Apr-2022 16:47:08.651 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.58]

log4j:ERROR Could not find value for key log4j.appender.luceneQuery

log4j:ERROR Could not instantiate appender named "luceneQuery".

26-Apr-2022 16:47:09.565 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

26-Apr-2022 16:47:09.709 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath

2022-04-26 16:47:09,879 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.13.5 [build 8703 based on commit hash da3c91d58b49066fe2f0052e452d9f35d99cdd6f] - synchrony version 4.0.0-master-bfb398fb

26-Apr-2022 16:47:09.888 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'

16:47:09 [Catalina-utility-2] o.a.c.c.C.[.[.[/synchrony-proxy][INFO] log - Initializing Spring DispatcherServlet 'dispatcher'

2022-04-26 16:47:11,783 WARN [Catalina-utility-1] [com.atlassian.config.ApplicationConfig] setApplicationHome Application home does not exist. Creating directory: /var/atlassian/application-data/confluence/shared-home

2022-04-26 16:47:11,787 INFO [Catalina-utility-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$saveSetupConfigIntoSharedHome$9 Writing setup configuration into shared home...

2022-04-26 16:47:11,794 INFO [Catalina-utility-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$saveSetupConfigIntoSharedHome$9 Finished writing setup configuration into shared home

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender (file:/opt/atlassian/confluence/confluence/WEB-INF/lib/org.apache.felix.framework-5.6.12.jar) to method java.net.URLClassLoader.addURL(java.net.URL)

WARNING: Please consider reporting this to the maintainers of org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

26-Apr-2022 16:47:17.531 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]

26-Apr-2022 16:47:17.540 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [8895] milliseconds

[Yevhen]

@Alexander Tchaikovski in your values.yaml do you have `confluence.clustering.enabled: true`? I don't see anything Hazelcast related in the logs.

For example, my logs have this:

2022-04-27 06:16:58,725 INFO [hz.confluence.event-1] [confluence.cluster.hazelcast.LoggingClusterMembershipListener] memberAdded [10.6.21.43]:5701 joined the cluster
2022-04-27 06:16:58,726 INFO [hz.confluence.event-1] [confluence.cluster.hazelcast.LoggingClusterMembershipListener] logClusterMembers Cluster now has 2 members: [[10.6.21.103]:5701, [10.6.21.43]:5701]
2022-04-27 06:16:59,739 INFO [Catalina-utility-1] [confluence.cluster.hazelcast.HazelcastClusterManager] startCluster Confluence cluster node identifier is [36bcb17f]
2022-04-27 06:16:59,751 INFO [Catalina-utility-1] [confluence.cluster.hazelcast.HazelcastClusterManager] startCluster Confluence cluster node name is [confluence-1]

[Alexander Tchaikovski]

i've managed to start the thing in cluster mode. pod-0 and pod-1 are there in config. one pod is marked in bold, i believe it's correct. there's another issue though. synchrony is not starting. reports:

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/atlassian/confluence/confluence/WEB-INF/packages/synchrony-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/atlassian/confluence/confluence/WEB-INF/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Starting synchrony

Exception in thread "main" clojure.lang.ExceptionInfo: error while starting state #'synchrony.config/config {:var #'synchrony.config/config}

any idea what it could be?

[Yevhen]

Hmmm... you must have hit a known issue with Synchrony. Can you please try this workaround?

[Alexander Tchaikovski]

thank you @Yevhen . very much! the issue was wrong synchrony url in the chart. i didn't place https:// in front of it.