Trusted Applications + Bouncy Castle + BigInteger = trouble?

[Sergey Svishchev]

Has anyone seen the condition when threads in an Atlassian app spin this way (eventually consuming all available CPU and/or thread pool)? If so, have you resolved it?

Doesn't seem to matter much where the TA library is called from; a good combination is JIRA + FishEye plugin.

java.lang.Thread.State: RUNNABLE
at java.math.BigInteger.mulAdd(BigInteger.java:1930)
at java.math.BigInteger.montReduce(BigInteger.java:1880)
at java.math.BigInteger.oddModPow(BigInteger.java:1850)
at java.math.BigInteger.modPow(BigInteger.java:1599)
at org.bouncycastle.crypto.engines.RSACoreEngine.processBlock(Unknown Source)
at org.bouncycastle.crypto.engines.RSABlindedEngine.processBlock(Unknown Source)
at org.bouncycastle.jce.provider.JCERSACipher.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2087)
at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.createEncryptedCertificate(BouncyCastleEncryptionProvider.java:229)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:54)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:67)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:58)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.execute(TrustedRequest.java:38)
at com.atlassian.jirafisheyeplugin.rest.FishEyeRestApiManagerImpl.callFisheye(FishEyeRestApiManagerImpl.java:175)
<...>

[Chad Barnes]

https://answers.atlassian.com/questions/11078895/very-poor-performance-of-jira-report-macros I believe we are having this problem. However, most our tools use the newer Trusted Apps. JIRA 6.3.8 Confluence 5.6.3 Fisheye 3.5.4 (Trusted Apps 3.0.3) Can one application link (even if not used when rendering a page) cause this problem?

[Sergey Svishchev]

I think Universe just hates me -- JIRA managed to wedge itself in another completely incomprehensible way :-(

[Guido Wischrop]

Thanks for the feedback!

[Sergey Svishchev]

Trusted Apps dev team fixed the issue and it's now up to each product's dev team to pick up the fix.

[Sergey Svishchev]

Yes -- it's been escalated to relevant dev team and is being tracked by them as https://ecosystem.atlassian.net/browse/TRUST-44.

It's quite hard to reproduce outside of production environment, since the trigger is not known yet.

Apparently, periodic thread dumps and the app's built-in profiler don't give enough data to pinpoint the root cause, and so using a 3rd party profiler seems to be the only way to find it (for JIRA, there's a YourKit plugin).

[Guido Wischrop]

OK, then it's not our JDK update. Thanks.

Did you get any feedback on your support ticket?

[Sergey Svishchev]

Java 1.7.0 update 25

[Guido Wischrop]

Our application link uses localhost, so the workaround would not work. Which JDK do you use?

[Sergey Svishchev]

Ah, finally a outside confirmation :) Could be useful for Atlassian (I logged a support case before going to Answers).

A workaround that seems to work is to stop incoming traffic from reaching the application while it starts up -- on frontend proxy/load balancer/firewall.

[Guido Wischrop]

This is our stack:

"pool-4966-thread-1" - Thread t@6468
   java.lang.Thread.State: RUNNABLE
	at java.math.BigInteger.mulAdd(BigInteger.java:1934)
	at java.math.BigInteger.squareToLen(BigInteger.java:1319)
	at java.math.BigInteger.oddModPow(BigInteger.java:1849)
	at java.math.BigInteger.modPow(BigInteger.java:1599)
	at org.bouncycastle.crypto.engines.RSACoreEngine.processBlock(Unknown Source)
	at org.bouncycastle.crypto.engines.RSABlindedEngine.processBlock(Unknown Source)
	at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown Source)
	at javax.crypto.Cipher.doFinal(Cipher.java:2087)
	at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.createEncryptedCertificate(BouncyCastleEncryptionProvider.java:229)
	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:54)
	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:67)
	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:58)
	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.execute(TrustedRequest.java:38)
[...]

   Locked ownable synchronizers:
	- locked <44d0babb> (a java.util.concurrent.ThreadPoolExecutor$Worker)

[Guido Wischrop]

Same problem here while Confluence 5.2.3 is trying to communicate with Jira 5.2.7. We did recently update our JDK from jdk1.7.0_25 to jdk1.7.0_51, could this be a problem?

We see also these exceptions:

ERROR [ajp-exec-14] [confluence.extra.jira.StreamableJiraIssuesMacro] logStreamableError Unable to render JIRA issues macro, connection to JIRA has been timeout.
 -- url: /wiki/display/xxxxxx | page: xxxxxxx | userName: xxxxxx | referer: xxxxxxx | action: viewpage
java.util.concurrent.TimeoutException
        at com.atlassian.confluence.extra.jira.StreamableJiraIssuesMacro$1.writeTo(StreamableJiraIssuesMacro.java:52)
        at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:251)
        at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:273)
        at com.atlassian.confluence.content.render.xhtml.view.ViewTableWrappingFragmentTransformer$1.writeTo(ViewTableWrappingFragmentTransformer.java:70)
        at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:251)
        at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:273)
        at com.atlassian.confluence.content.render.xhtml.storage.StorageXhtmlTransformer.transform(StorageXhtmlTransformer.java:44)
        at com.atlassian.confluence.content.render.xhtml.TransformerChain.transform(TransformerChain.java:41)
        at com.atlassian.confluence.content.render.xhtml.PluggableTransformerChain.transform(PluggableTransformerChain.java:53)
        at com.atlassian.confluence.content.render.xhtml.DefaultRenderer.render(DefaultRenderer.java:80)