Confluence behind apache reverse proxy denies Collaborate

[M_ Ludwig]

Hi Community,

we are running our Confluence behind an apache https proxy on a Debian9 Server. Confluence is the only app running on that machine.

However Collaborate stops working and can not reach its backend.

In the apache logs i can see following entries:

[proxy:error] [pid 7851] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8091 (localhost) failed

Port 8091 is up and running, but somehow the websocketconnect fails miserably.

Here is our Apache Config for the proxy:

ApacheConfig SSL for Confluence

#### confluence section
<VirtualHost *:443>
TimeOut 1000
ServerName confluence.our.site
DirectoryIndex index.html

SSLEngine on
#Certificates
SSLCertificateFile /etc/apache2/ssl/our.site.crt
SSLCertificateChainFile /etc/apache2/ssl/AlphaSSL_Intermediate_CA.pem
SSLCertificateKeyFile /etc/apache2/ssl/our.site.key
SSLCACertificateFile /etc/apache2/ssl/GlobalSign_Root_R1_CA.pem

SSLProxyEngine on
RewriteEngine on
ProxyRequests off
ProxyPreserveHost on

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"

LogLevel info ssl:debug
ErrorLog ${APACHE_LOG_DIR}/our.site-ssl-error.log
CustomLog ${APACHE_LOG_DIR}/our.site-ssl-access.log combined

<Proxy *>
Require all granted
</Proxy>

ProxyPass /synchrony http://localhost:8091/synchrony
<Location /synchrony>
Require all granted
RewriteEngine on
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]

RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteRule .* ws://localhost:8091%{REQUEST_URI} [P]
</Location>

ProxyPass / http://localhost:8090/
ProxyPassReverse / http://localhost:8090/

<Directory "/var/www/our.site">
Options +FollowSymLinks +SymLinksIfOwnerMatch
AllowOverride All
Require all granted
</Directory>

#Certbot
ProxyPass /.well-known !
ProxyPassReverse /.well-known !

#additional ssl
SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on
SSLCompression off

</VirtualHost>
###end of confluence section

#### balsamiq rtc section
Listen our.site:9083
<VirtualHost *:9083>
SSLEngine on
#Certificates
SSLCertificateFile /etc/apache2/ssl/our.site.crt
SSLCertificateChainFile /etc/apache2/ssl/AlphaSSL_Intermediate_CA.pem
SSLCertificateKeyFile /etc/apache2/ssl/our.site.key
SSLCACertificateFile /etc/apache2/ssl/GlobalSign_Root_R1_CA.pem

ProxyRequests off
ProxyPreserveHost On
RewriteEngine on

<Proxy *>
Require all granted
</Proxy>

ProxyPass / http://localhost:9083/
RewriteEngine on
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteRule .* ws://localhost:9083%{REQUEST_URI} [P]
</VirtualHost>
###end of balsamiq rtc section

We also have a port 80 vhost which redirects to the https:

<VirtualHost *:80>
TimeOut 1000
UseCanonicalName On
ServerAdmin root@localhost
ServerName confluence.our.site
DocumentRoot "/var/www/our.site"
DirectoryIndex index.html
ProxyRequests Off

LogLevel info
ErrorLog ${APACHE_LOG_DIR}/our.site-prod-error.log
CustomLog ${APACHE_LOG_DIR}/our.site-access.log combined

<Directory "/var/www/our.site">
AllowOverride All
Options +FollowSymLinks +SymLinksIfOwnerMatch
Require all granted
</Directory>

RewriteEngine on
#RewriteCond %{THE_REQUEST} !/.well-known/acme-challenge/ [NC]
RewriteCond %{SERVER_NAME} =confluence.solar-log.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

our confluence server.xml:

<Server port="8000" shutdown="SHUTDOWN" debug="0">
<Service name="Tomcat-Standalone">

<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
address="127.0.0.1"
maxThreads="48" minSpareThreads="10"
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"
scheme="https" secure="true" proxyName="confluence.our.site" proxyPort="443"/>

<Engine name="Standalone" defaultHost="localhost" debug="0">
<Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">
<Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">
<!-- Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties -->
<Manager pathname=""/>
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/>
</Context>

<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0"
reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/>
</Context>
</Host>
</Engine>
</Service>
</Server>

We are unable to open the ws sockets - this is also for balsamiq which also does not work.

In order to use confluence we had to stop collaborate completely. But this is not what we wish. We need collaborate working.

We have no running firewall on that server, but we have security tools rkhunter, aide, auditd, acct and lynis working.

 Can please someone help?

Greetings,

    Tux

[M_ Ludwig]

Hi Alexis,

thank you for response.

Yes i assure that Port 8091 is reachable via telnet:

telnet localhost 8091
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^[9
HTTP/1.1 414 Request-URI Too Long

text is emptyConnection closed by foreign host.

[M_ Ludwig]

Hi there,

we still have those severe problems which caused us to stop collaborate and thus balsamiq also is not working since it also uses websockets over synchrony.

Telnetting to the machines synchrony port works just fine from the local machine and also from a distant machine.

Also a wget of the heartbeat which works over synchrony is working and getting a file back with heartbeat ok.

But if we start Collaborate and thus restarting syncrony, we see in the apache ssl error logs always the following entries:

[proxy:error] [pid 17880] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:8091 (*) failed

[proxy_wstunnel:error] [pid 17880] [client 192.168.110.218:50896] AH02452: failed to make connection to backend: localhost

[proxy:error] [pid 19466] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8091 (localhost) failed

[proxy_http:error] [pid 19466] [client 192.168.110.218:50892] AH01114: HTTP: failed to make connection to backend: localhost

For a test we installed confluence on another server (also Debian9 with same settings) with exactly the same configuration in confluence itself and apache - it works there like a charm. On our productive server we always see above nasty errors we can not find out where they are coming from.

 Any help in that is appreciated.

Greetings,

    Tux

[Alexis Robert]

Hi @M_ Ludwig , 

I have pretty much the same configuration (my Apache proxy runs on another VM but very similar otherwise). Here's my Apache config (I left out the SSL stuff) :

<Proxy *>
Require all granted
</Proxy>
ProxyPass /synchrony http://my.ip:8091/synchrony

<Location /synchrony>
Require all granted
RewriteEngine on
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteRule .* ws://my.ip:8091%{REQUEST_URI} [P]
</Location>

ProxyPass / http://my.ip:8090/
ProxyPassReverse / http://my.ip:8090/

<Location />
Require all granted
</Location>

Did you make sure that you're not blocking requests on 8091 port ? Are you able to telnet to that port for example ?

[Alexis Robert]

Hi @M_ Ludwig , 

if you keep seeing the errors on your production servers, but the setup works fine on your test server, then the issue is definitely with your production server. 

I would check again if there is not a SELinux or iptables policy preventing the connection to synchrony.

[M_ Ludwig]

Hi Alexis,

we have no iptables set up and no selinux running. SE is not even installed on the server.

The problem is also only located to Apache proxy/Synchrony both running on exactly this machine having problems to connect via localhost or 127.0.0.1.

Connecting to the synchrony port on localhost from a login shell of the server itself is working as a charm.

We checked everything twice and could not find a configuration different from our testsystem. This exactly is the reason why we are asking here - we simply have no idea where to search next.

Greetings,

   Tux