Development Panel does not work through proxy

[Ted W]

The AppLink configuration between Stash and JIRA appears to create without issue. Both systems use mod_proxy through an Apache reverse proxy to translate the external address https://server.example.com to https://server-backend.example.org. On each of the back end systems mod_proxy, again, translates https://server-backend.example.org to http://server-backend.example.org:8080 (in the case of Jira) and http://server-backend.example.org:7990 (in the case of Stash).

With the Applications configured this way and the AppLink configured to use the external addresses, the Link works as expected everywhere except on issues where the Development Panel would normally show the Commit/Pull request information on the right hand side. Instead it is showing the "Source" tab and the information in the tab shows as expected.

If I configure the servers to run outside of the proxy (e.g. you connect directly to http://server-backend.example.org:8080) and I create the AppLink using the direct, non-ssl, non-proxy address, the Development panel functions as expected. Below I've provided the the <Connector> portion of each servers server.xml file as well as the apache configurations for each system on the reverse proxy server.

JIRA server.xml:

<Connector port="8080"
                   scheme="https"
                   proxyPort="443"
                   proxyName="jira.example.com"

                   maxThreads="150"
                   minSpareThreads="25"
                   connectionTimeout="20000"

                   enableLookups="false"
                   maxHttpHeaderSize="8192"
                   protocol="HTTP/1.1"
                   useBodyEncodingForURI="true"
                   redirectPort="8443"
                   acceptCount="100"
                   disableUploadTimeout="true"/>

Stash server.xml:

<Connector port="7990" protocol="HTTP/1.1"
                   connectionTimeout="20000"
                   useBodyEncodingForURI="true"
                   redirectPort="8443"
                   compression="on"
                   compressableMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript,application/x-javascript"
                   secure="true"
                   scheme="https"
                   proxyName="stash.example.com"
                   proxyPort="443"/>

JIRA apache configuration:

NameVirtualHost server.example.com:443
<VirtualHost server.example.com:443>
ServerName server.example.com
ErrorLog logs/server_ssl_error_log
TransferLog logs/server_ssl_access_log
LogLevel warn
SSLEngine on
SSLProtocol all -SSLv2
SSLHonorCipherOrder On
SSLCipherSuite -All:EDH-DSS-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:DES-CBC3-SHA:RC4-MD5:RC4-SHA:DHE-DSS-AES128-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:AES128-SHA:AES256-SHA
SSLCertificateFile /etc/pki/tls/certs/server.crt
SSLCertificateKeyFile /etc/pki/tls/private/server.key
SSLCertificateChainFile /etc/pki/tls/certs/incommon.crt
SSLCACertificateFile /etc/pki/tls/certs/bundle.crt
SSLVerifyClient optional
ErrorDocument 403 "You are not allowed to access this site.  Please ensure you are on a trusted network"
<Proxy *>
    Order deny,allow
    Deny from all
    # All private subnets
    Allow from 10.0.0.0/8
    # All MALNET VPN
    Allow from xxx.xxx.xxx.xxx
    # All TESTNET VPN
    Allow from xxx.xxx.xxx.xxx
    # All Workstations
    Allow from xxx.xxx.xxx.0/24
    # All DEV IPs
    Allow from xxx.xxx.xxx.0/24
</Proxy>

<Location "/">
    Order deny,allow
    Deny from all
    # All private subnets
    Allow from 10.0.0.0/8
    # All MALNET VPN
    Allow from xxx.xxx.xxx.xxx
    # All TESTNET VPN
    Allow from xxx.xxx.xxx.xxx
    # All Workstations
    Allow from xxx.xxx.xxx.0/24
    # All DEV IPs
    Allow from xxx.xxx.xxx.0/24
</Location>
SSLProxyEngine On
ProxyRequests On
ProxyPreserveHost On
ProxyPass /cert_info.php !
ProxyPass / https://10.xxx.xxx.228:443/ retry=0 connectionTimeout=5 timeout=300
ProxyPassReverse / http://10.xxx.xxx.228:443/ retry=0
ProxyTimeout 300
# Only require certificates for some projects
<LocationMatch "^/(git/projects/IT|reviews|git/projects/XXX|git/projects/xxx|users)">
SSLOptions +StdEnvVars
RewriteEngine On
# git (fisheye triggers)
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.80$
# spud Is this still needed?
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.109$
# wiki
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.167$
# wiki-be
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.184$
# jira-be
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.12$
# scm-be
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.228$
RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.252$
RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.253$
RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.251$
# xxxxxxxxx for jenkins
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.68$
RewriteCond %{SSL:SSL_CLIENT_VERIFY} !^SUCCESS$
RewriteRule .* /cert_info.php [R,L]
</LocationMatch>
<Location /cert_info.php>
SSLOptions +StdEnvVars
</Location>
SSLVerifyDepth  10
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>
SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</VirtualHost>

JIRA Apache Config:

NameVirtualHost jira.example.com:443
SSLStrictSNIVHostCheck off
<VirtualHost jira.example.com:443>
ServerName jira.example.com
ErrorLog logs/ssl_error_log
TransferLog logs/ssl_access_log
LogLevel warn
SSLEngine on
SSLProtocol all -SSLv2
SSLHonorCipherOrder On
SSLCipherSuite -All:EDH-DSS-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:DES-CBC3-SHA:RC4-MD5:RC4-SHA:DHE-DSS-AES128-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:AES128-SHA:AES256-SHA
SSLCertificateFile /etc/pki/tls/certs/jira.crt
SSLCertificateKeyFile /etc/pki/tls/private/jira.key
SSLCertificateChainFile /etc/pki/tls/certs/incommon.crt
SSLCACertificateFile /etc/pki/tls/certs/bundle.crt
SSLVerifyClient optional
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>

<Location "/">
   Order deny,allow
   Deny from all
   Allow from all
</Location>
SSLProxyEngine On
ProxyRequests On
ProxyPreserveHost on
# Jira-be
ProxyPass /cert_info.php !
ProxyPass / https://10.xxx.xxx.12:443/ retry=0
ProxyPassReverse / http://10.xxx.xxx.12:443/ retry=0
ProxyTimeout 300
<LocationMatch "^/(?!cert_info.php)">
SSLOptions +StdEnvVars
RewriteEngine On
# spud
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.109$
# wiki
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.167$
# wiki-be
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.184$
# jira-be
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.12$
RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.251$
RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.252$
RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.253$
# scm-be
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.228$
# RT # 3790 allow xxxxxxx-xxxxxx
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.95$
# Jama
RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.192$
RewriteCond %{SSL:SSL_CLIENT_VERIFY} !^SUCCESS$
RewriteRule .* /cert_info.php [R,L]
</LocationMatch>
<Location /cert_info.php>
SSLOptions +StdEnvVars
</Location>
SSLVerifyDepth  10
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>
SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</VirtualHost>

I have my suspicions as to why this is not working but I'm not familiar enough with how the two applications communicate over the AppLink connection to know where to begin. There are no obvious errors in either the Stash or JIRA logs and all other features of the AppLink appear to work just fine. Any suggestions would be greatly appreciated.

[Dave C]

It's possible you could be running into the bug we have tracked in https://jira.atlassian.com/browse/JRA-40968 - there are a number of workarounds listed in there you could try.

[Ted W]

Quick update. I re-configured the proxy to use port 80 and disabled ssl for Stash, leaving JIRA to proxy over https and the development panel appears to work now.... Still not sure why this wouldn't work over an https reverse proxy.

Once I figured the above information out I re-enabled https on Stash. I then created a p12 file from the stash server certificate, key and CA root and imported that as alias "tomcat" in /usr/lib/jvm/java/jre/lib/security/cacerts, ensured that the proxy server also had the proper cert/key and CA specified in the apache config and none of this seemed to resolve the issue. Any other ideas why ssl proxy would be causing the development panel to malfunction would be appreciated.

[Ted W]

I just checked and no, it does not change the behavior. I also tried setting up the "Application Link" using the direct http://jira.example.org:8080 and http://stash.example.org:7990/git urls, leaving the respective applications' Base URL set to the proxy URL and it connected ok, once again and the Development panel now shows up with a message: "JIRA is having difficulty contacting Stash. If this condition persists, please contact your JIRA administrators." and the Source tab is also visible with the message: "Request to https://stash.example.com/git failed: Stash returned an error: 401 Unauthorized" which I found odd... not sure if it's useful information or not, though.

[Boris Berenberg]

Does it work if you use a 1 stage proxy config instead of 2?