Communications breakdown error on workflow transition

[adamtwarner94]

We are experiencing an intermittent "Communications Breakdown" error on our JIRA instance when trying to transition issues through the workflow, and I am wondering if anyone can offer us a hint on where to look or how best to troubleshoot.   The error occurs on the AJAX call to WorkflowUIDispatcher.jspa, which gets an empty response, triggering the "Communications Breakdown - The JIRA server could not be contacted" error dialog.  The error comes up almost immediately after pressing a workflow button - no time lag typical of network timeout.

The error is always cleared up by refreshing the page.  It only seems to happen on the workflow dispatch on the view issue page, not other AJAX calls or anywhere else in the app.  And it seems to occur only after you've been idle on the page for about a minute or more, which would seem to indicate some sort of network timeout...but it's peculiar that it only happens on that call, not other AJAX calls on the page.    

On Chrome (v46) console the error is "net::ERR_CONNECTION_ABORTED".  On IE (11), console error is "XMLHttpRequest: Network Error 0x2eff, Could not complete the operation due to error 00002eff."    Searching the internet about these errors yields about 100 different possible causes. 

More trace info from Chrome:

net::ERR_CONNECTION_ABORTED

send	@	batch.js:167
d.extend.ajax	@	batch.js:161
JIRA.SmartAjax.makeRequest	@	batch.js:379
JIRA.Dialog.AJS.Control.extend._contentRetrievers.ajax	@	batch.js:2466
JIRA.Dialog.AJS.Control.extend._setContent	@	batch.js:2466
JIRA.FormDialog.JIRA.Dialog.extend._setContent	@	batch.js:2472
(anonymous function)	@	batch.js:367
JIRA.Dialog.AJS.Control.extend._show	@	batch.js:2466
JIRA.Dialog.AJS.Control.extend.show	@	batch.js:2466
(anonymous function)	@	batch.js:2658
d.event.dispatch	@	batch.js:74
g.handle.i	@	batch.js:66

On Firefox, same error: 

ajax[44238426] error : {
successful  : false,
status      : 0,
statusText  : error,
hasData     : false,
readyState  : 0,
requestId   : 44238426,
aborted     : undefined,
}

batch.js:195:345

I do not think this is any issue with JIRA itself - I've determined that the workflow dispatch call is not making it to JIRA - at least there is no trace in the HTTP access log.   Likely it is some sort of network issue since we run JIRA in a complex enterprise network environment:   SSO with custom authenticator under a websphere portal, with the portal at one data center and JIRA at another, communicating on commodity internet over SSL, with firewalls at both ends, and SSL offloading through a load balancer and reverse proxy on the JIRA end.  (Only one JIRA instance, though - load balancer just does SSL offloading and passes the traffic through, no actual load balancing.)     The error only happens when running through the portal / WAN connection; it does not happen when accessing JIRA directly from an internal network connection (using a hosts file entry to simulate the portal for reverse proxy).

JIRA has run fine in this configuration for three years, and the errors suddenly started occurring a couple of weeks ago...but due to the complexity of the environment we've been unable to determine exactly what changed.  We didn't change anything with JIRA itself, or its server.

Anyone have any insight to share?  Is there something different about the call to WorkflowUIDispatcher that might give us a clue where to look?

As a workaround, would it be possible/advisable to modify the JIRA source code to refresh the page before issuing the workflow dispatch?  (We run v 6.2.6)

Any guidance greatly appreciated.... 

[adamtwarner94]

Just thought I'd post a follow up on this for posterity, in case anyone else ever runs into the same strange issue.  We were able to work around this issue by patching JIRA\atlassian-jira\WEB-INF\classes\com\atlassian\jira\security\xsrf\SimpleXsrfTokenGenerator.class  to use "_" as a delimiter in the atl_token param instead of "|".  For some weird reason, something in our network did not like the pipes and it closed the network connection, and you then had to refresh the page and perform the workflow action withing the HTTP Keep Alive (about 15 secs).   We will probably never figure out the true root cause, but this got us past it.

[Martin Ross]

This is due to the fact that the pipe character should be URL encoded..

GET https://jira.xxx.com/secure/WorkflowUIDispatcher.jspa?id=67800&action=…5a02f8ed33ef9143eef123891|lin&decorator=dialog&inline=true&_=1457391270578 net::ERR_CONNECTION_RESETsend @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:178d.extend.ajax @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:172SmartAjax.makeRequest @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:617Control.extend._contentRetrievers.ajax @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:2543Control.extend._setContent @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:2543Dialog.extend._setContent @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:2614(anonymous function) @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:599Control.extend._show @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:2543Control.extend.show @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:2543(anonymous function) @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:2890d.event.dispatch @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:85g.handle.i @ batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:77
batch.js?atlassian.aui.raphael.disabled=true&locale=en-US:216 ajax[64470578] error : {
successful : false,
status : 0,
statusText : error,
hasData : false,
readyState : 0,
requestId : 64470578,
aborted : undefined,
}

However if we replace the pipes with URL encoded character %07C

https://jira.xxx.com/secure/CommentAssignIssue!default.jspa?atl_token=B58G-RLM5-6JZY-K8RV%7C3e22cf27a1058d35a02f8ed33ef9143eef123891%7Clin&_=1457391123988&decorator=dialog&id=67800&action=5&inline=true

it works...

I guess this is a bug with JIRA since it really should be escaping the pipe character with URL encoding...