PKIX path building fails when running any REST API call from Jira DC to Jira Cloud

[Ross McCoy]

Hello,

I have two instances of Jira: instance A is a Jira Cloud instance, and instance B is a Jira Data Center instance running 7.13.1 with 2 nodes.

I am trying to run a REST API call from Instance B to Instance A using the following script fragment:

def baseURL = "https://MYSITE.atlassian.net/rest/api/3/issue/ISSUE-KEY/transitions"
URL url = new URL(baseURL);
def body = ["transition" : ["id" : "71"]];

URLConnection connection = url.openConnection();
connection.setRequestMethod("POST"); 
connection.doOutput = true
connection.setRequestProperty("Content-Type", "application/json;charset=UTF-8")
connection.setRequestProperty("Authorization", "Basic $encoded")
connection.outputStream.withWriter("UTF-8") { new StreamingJsonBuilder(it, body) } 
connection.connect();

However, this script fails to run with the following error:

There was an error during the execution of your script against issue ISSUE-KEY

Message:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Stack:

sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1964)
sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:298)
script15823128842791779017250.run(script15823128842791779017250.groovy:57)

We get the same error regardless of whether we run the REST API call using the JMWE script console, the Scriptrunner script console, or using a scripted postfunction with either add-on. Myself and the other JIRA administrator (and nearly all users) have consistently experienced this error, however on one or two occasions it seems that the REST API call has succeeded for another user when they transitioned an issue with a scripted postfunction that contains a REST API call, but this success is not reproducible and there seems to be no commonality between when it fails vs the very rare cases when it succeeds.

Additionally, I am experiencing this error with ALL REST API calls going from Instance B (Jira DC) to Instance A (Jira Cloud), not just the one above. Even the most basic GET REST API request (GET /rest/api/3/issue/{issueIdOrKey}) produces the error above. We are using Okta for our SSO solution, and the account we are authenticating the REST API request as is configured to bypass Okta.

However, there are no problems when we run a REST API call in the opposite direction (Cloud->Server). There is a single reciprocal application link between both instances that shows a status of "Connected" when viewed in the Application Links admin page.

To troubleshoot this error, we followed the steps listed on this page:
https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html

On both nodes in Instance B (Jira DC), we downloaded and ran SSLPoke.class, using both the system Java and the Java install that Jira is using, and SSLPoke.class reported a successful connection on both nodes with both versions of Java.

The next step we took was to run the following cURL command on the command line of either node:

curl --request POST \
--url 'https://MYSITE.atlassian.net/rest/api/3/issue/IISSUE-KEY/transitions' \
--user 'USERNAME:API_TOKEN' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{"transition": {"id": "71"}}'

This cURL command completed successfully and the issue was transitioned to the appropriate status.

What are our next troubleshooting steps from here? It doesn't seem to make sense that, if there was a problem with the certificate store, that there would ever be a successful REST API call run between Jira DC and Jira Cloud, but that is indeed the (very rare and unfortunately unreproducible) case.

Addendum:

I tried the same REST API calls as above from a different Jira Server instance hosted outside of the network in question, and the REST API call from Jira Server-> Jira Cloud completed successfully.

[Thomas Deiler]

Dear @Ross McCoy ,

I have the JVM of the calling Jira Server under suspect. It looks like B cannot validate the SSL certificate of A.

I don't believe that the Atlassian Cloud has invalid certs. Eventually the JVM of B is outdated, means, it does not have the root certificate in its build keystore, that was used to sign SSL cert of A.

Solution 1: Renew the JVM of B.

Solution 2: Manually add the certificate of A to the keystore of B.

So long

Thomas