Confluence Synchrony not working behind Netscaler ADC

[Andrew Downs]

Hi guys, I have configured Confluence behind a Netscaler ADC but I cannot seem to get Synchrony working behind the load balancer regardless of what i do. Web sockets are enabled, and this is working as we have a few services published though the same service that use web sockets and that works.

If I access the server on http://confluence.domain.com:8090 everything is happy, but when I access through the Netscaler as confluence.domain.com synchrony will not turn on, therefore editing KA’s is limited.

However if I try https://confluence.domain.com/Synchrony I get an OK returned? Surely this means it’s working and should be turning on?

Am I missing something here? I really don’t want to go the NGINX route just to get this working.

I am not using a content path for confluence.

[Daniel Stoeber]

Hi,

I had the same issue. Whean I had my Confluence behind a vpn vserver it was not working.

Client -> VPN-vServer -> vLB -> Service

After switching my Citrix ADC Config to an cs vserver and an HTTP-Netprofile with websocks enabled on the Loadbalancing ServiceGroup its now working.

Client -> CS-vServer -> vLB -> Service

Greetings

Daniel

[Andrew Downs]

Hi Daniel,

Sorry for the late response, had to focus on more pressing projects.

I have a net profile configured allowed websockets, and it works for other applications.

I have configured Confluence as Content Switching Virtual server, my policy is as follows:

HTTP.REQ.HOSTNAME.EQ("confluence.contoso.com")

HTTP.REQ.HOSTNAME.EQ("confluence.contoso.com") && HTTP.REQ.URL.CONTAINS("synchrony")

I have bound the Load Balancing virtual server (Non Addressable) to a Load Balancing service group.

Both the Load Balancing Virtual Server and Service Group have the Websockets HTTP profile configured.

Did you use a non addressable object or a object with the IP address of the CS?

[Andrew Downs]

Hi @Edwin Kyalangalilwa

Thanks for the response. 

1) Confluence is 6.10.0, I also tried to upgrade 6.15.2. However what is weird is that synchrony never worked 100% before the upgrade, I could at least start it. After the upgrade it didn’t start at all, even reverting to the snapshot I have taken before the upgrade doesn’t work

2) ProxyName is configured as well as port 443 as the proxy port

2.1) not using any context path for confluence 

3 & 4) Not by my laptop at the moment, will revert once I get a chance 

i did try configure Nginx, but also not to sure if I am configuring it right. I did a clean install of NGINX and edited the default.conf in /etc/NGINX/conf.d, but not convinced I full understand that configuration. 

I dont want want to use a context path for confluence. If it’s simpler on NGINX then I’ll use NGINX, I am just at a stage where I need to get it working and allow users to work with the collaborative editing. 

Regards

[Andrew]

Hi @Andrew Downs ,

For NGINX you can find info here https://confluence.atlassian.com/conf63/running-confluence-behind-nginx-with-ssl-929729913.html

B.R.

[Andrew Downs]

Good morning @Andrew and @Edwin Kyalangalilwa

Please see below for a snippet from the atlassian-synchrony.log when attempting to enable Synchrony:

2019-04-23 06:04:57,451 DEBUG [176:StdErrHandler [/opt/atlassian/confluence/jre/bin/java]] WARNING: boolean? already refers to: #'clojure.core/boolean? in namespace: medley.core, being replaced by: #'medley.core/boolean?
2019-04-23 06:04:57,559 DEBUG [176:StdErrHandler [/opt/atlassian/confluence/jre/bin/java]] WARNING: Inst already refers to: #'clojure.core/Inst in namespace: schema.core, being replaced by: #'schema.core/Inst
2019-04-23 06:05:00,940 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] Running QueryDSL sytem...
2019-04-23 06:05:01,039 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"message":"synchrony.querydsl.jdbc [info] connection pool test-connection-on-checkin: true idle-connection-test-period: 100 max-idle-time-excess-connections: 100 max-idle-time: 3600","ns":"synchrony.querydsl.jdbc","level":"info"}}
2019-04-23 06:05:01,053 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] MLog clients using slf4j logging.
2019-04-23 06:05:01,065 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] Initializing c3p0-0.9.5.1 [built 16-June-2015 00:06:36 -0700; debug? true; trace: 10]
2019-04-23 06:05:01,120 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"message":"synchrony.event-bus [info] cluster membership transition","transition":{"type":"join","member":{"member-id":"local","host":null,"local?":true}},"ns":"synchrony.event-bus","level":"info"}}
2019-04-23 06:05:04,185 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/heartbeat","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}
2019-04-23 06:05:04,241 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/jwt-key","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}
2019-04-23 06:05:19,013 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/heartbeat","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}
2019-04-23 06:05:49,001 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/heartbeat","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}
2019-04-23 06:06:19,000 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/heartbeat","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}
2019-04-23 06:06:49,032 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/heartbeat","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}
2019-04-23 06:07:19,000 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/heartbeat","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}
2019-04-23 06:07:49,000 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/heartbeat","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}
2019-04-23 06:08:19,026 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/heartbeat","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}
2019-04-23 06:08:49,000 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/heartbeat","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}
2019-04-23 06:09:19,002 DEBUG [177:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] {"synchrony":{"request":{"remote-addr":"127.0.0.1","uri":"/synchrony/heartbeat","request-method":"get","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}}

Below is the configuration for server.xml:

<Server port="8000" shutdown="SHUTDOWN" debug="0"><Service name="Tomcat-Standalone"><Connector port="8090" connectionTimeout="20000" redirectPort="8443"maxThreads="48" minSpareThreads="10"enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"protocol="org.apache.coyote.http11.Http11NioProtocol"proxyName="confluence.domain.com" proxyPort="443" scheme="https"SSLEnabled="false"/>

<Engine name="Standalone" defaultHost="localhost" debug="0">

<Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4"><Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true"><!-- Logger is deprecated in Tomcat 5.5. Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties --><Manager pathname="" /><Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" /><Resource name="jdbc/confluence" auth="Container" type="javax.sql.DataSource"username="redacted"password="redacted"driverClassName="com.microsoft.sqlserver.jdbc.SQLServerDriver"url="jdbc:sqlserver://redacted:1433;instanceName=redacted;DatabaseName=redacted"maxTotal="60"maxIdle="20"validationQuery="SELECT 1" /></Context>

<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0" reloadable="false" useHttpOnly="true"><Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" /></Context></Host>

</Engine>

<!--To run Confluence via HTTPS:* Uncomment the Connector below* Execute:%JAVA_HOME%\bin\keytool -genkey -alias tomcat -keyalg RSA (Windows)$JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA (Unix)with a password value of "changeit" for both the certificate and the keystore itself.* Restart and visit https://localhost:8443/

For more info, see https://confluence.atlassian.com/display/DOC/Running+Confluence+Over+SSL+or+HTTPS--><!--<Connector port="8443" maxHttpHeaderSize="8192"maxThreads="150" minSpareThreads="25"protocol="org.apache.coyote.http11.Http11NioProtocol"enableLookups="false" disableUploadTimeout="true"acceptCount="100" scheme="https" secure="true"clientAuth="false" sslProtocols="TLSv1,TLSv1.1,TLSv1.2" sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2" SSLEnabled="true"URIEncoding="UTF-8" keystorePass="<MY_CERTIFICATE_PASSWORD>"/>--></Service></Server>

The error I get when attempting to enable Synchrony is as follows:

Something went wrong

Please try again.

When running the Chrome Developer Tools I do see it trying to get to the following:

batch.js?healthcheck-resources=true&hostenabled=true&locale=en-US&whisper-enabled=true:140 Uncaught TypeError: Cannot read property 'taskId' of null
at Object.<anonymous> (batch.js?healthcheck-resources=true&hostenabled=true&locale=en-US&whisper-enabled=true:140)
at k (batch.js?locale=en-US:203)
at Object.fireWith [as resolveWith] (batch.js?locale=en-US:204)
at c (batch.js?locale=en-US:329)
at XMLHttpRequest.b (batch.js?locale=en-US:341)
(anonymous) @ batch.js?healthcheck-resources=true&hostenabled=true&locale=en-US&whisper-enabled=true:140
k @ batch.js?locale=en-US:203
fireWith @ batch.js?locale=en-US:204
c @ batch.js?locale=en-US:329
b @ batch.js?locale=en-US:341
XMLHttpRequest.send (async)
XMLHttpRequest.send @ batch.js?healthcheck-resources=true&hostenabled=true&locale=en-US&whisper-enabled=true:719
send @ batch.js?locale=en-US:340
ajax @ batch.js?locale=en-US:334
ajax @ batch.js?healthcheck-resources=true&hostenabled=true&locale=en-US&whisper-enabled=true:105
c @ batch.js?healthcheck-resources=true&hostenabled=true&locale=en-US&whisper-enabled=true:130
g @ batch.js?healthcheck-resources=true&hostenabled=true&locale=en-US&whisper-enabled=true:130
A @ batch.js?healthcheck-resources=true&hostenabled=true&locale=en-US&whisper-enabled=true:140
dispatch @ batch.js?locale=en-US:248
h @ batch.js?locale=en-US:239
batch.js?healthcheck-resources=true&hostenabled=true&locale=en-US&whisper-enabled=true:719 GET https://confluence.domain.com/rest/synchrony-interop/synchrony-status net::ERR_NETWORK_CHANGED

Not sure what that means I suspect it may be something with the Netscaler, but I am not sure what would have changed.

I will try the nginx route and see if I can get that working as well just to rule out the Netscaler.

[Andrew]

Hi @Andrew Downs ,

Maybe,

Don't remove or comment out the http connector, as the Synchrony proxy health check, still requires HTTP. If you don't want to include the http connector, you can use the synchrony.proxy.healthcheck.disabled system property to disable the health check. 

https://confluence.atlassian.com/doc/running-confluence-over-ssl-or-https-161203.html

B.R.

[Andrew Downs]

Hi @Andrew 

Thank you, I have for now configured Confluence to user the NGINX proxy and so far seems to be ok. However I still need to set Synchrony Collaborative editing to Limited for Confluence to work, this is not the end state I want to be in, but for now it is working. I will be deploying a new Confluence Development Instance so I can test more without affecting our production users.

I will try leave the HTTP listener enabled as well.

[Edwin Kyalangalilwa]

Hi @Andrew Downs ,

Please provide some more information.

• Confluence version
• Is proxyName and proxyPort configured in server.xml?
• Synchrony error log: atlassian-synchrony.log 
• Error in the browser when attempting to create or edit a page.