Update to BitBucket server 7.1.0 push never ends

[Dionisios]

Hello, I've updated bb server to 7.1.0 (windows) and I can't make a push. On the client side process seems running forever.

C:\Users\Administrator\Desktop\Android>git push --verbose
Pushing to https://XXXXXX.git
Enumerating objects: 181, done.
Counting objects: 100% (142/142), done.
Delta compression using up to 4 threads
Compressing objects: 100% (98/98), done.
Writing objects: 100% (101/101), 50.97 KiB | 1.96 MiB/s, done.
Total 101 (delta 31), reused 0 (delta 0), pack-reused 0
POST git-receive-pack (52353 bytes)

On the server side I've noticed the following error and also an always running git-http-backend.exe process causing high cpu load.

2020-03-30 20:10:46,072 ERROR [https-jsse-nio-7990-exec-3] admin @1C78VOBx1206x3x0 97.70.122.90 "POST /scm/EUR/Myrepository.git/git-receive-pack HTTP/1.1" c.a.s.i.s.g.p.h.GitSmartExitHandler EUR/myrepository[53]: Write request from 97.70.122.90 failed: com.atlassian.utils.process.ProcessException: Non-zero exit code: 1

I've tried from 3 different clients with the same result. Can someone help me?

[Bryan Turner]

We've received several support cases for this issue recently, and we've been investigating it internally. We've created BSERV-12285 to track the issue, and based on what we've been able to reproduce so far we believe the root cause is a regression in Tomcat 9.0.31. Since this appears to be caused by a bug in Tomcat, there's no workaround for the issue. We're working on releasing Bitbucket Server 7.1.1 with Tomcat 9.0.33, which includes a fix for the regression and which we believe will resolve this issue.

Best regards,
Bryan Turner
Atlassian Bitbucket 

[Andrew Bilukha]

Why didn't you pull this version off of the available releases? We just upgraded this past Friday to 7.1.0 completely unaware of showstopper CPU load issues. 
This morning we already had to add CPU's to the VM, and RAM.  This is a very poor customer communication -- you were supposed to warn all your license payers of this show stopper immediately when you have learned about it.

[Bryan Turner]

@Andrew Bilukha,

Can you share how you came to download 7.1.0 instead of 7.1.1? The download page defaults to the newest release, which is 7.1.1. Since 7.1.1 was released, the only way to download 7.1.0 would be to do so explicitly. If the download page was still defaulting to 7.1.0 when you downloaded it, that means 7.1.1 with the fix was not yet available.

(Note: I'm not challenging anything you've written. I just want to clarify how you ended up downloading 7.1.0 instead of 7.1.1.)

Best regards,
Bryan Turner
Atlassian Bitbucket

[Andrew Bilukha]

Greetings Brian,

I downloaded it on 4/3, ran t he upgrade in DEV environment, waited for a week running it in DEV, and then installed in Prod on 4/10.

[Dionisios]

I had to make it work immediately so I went back to my previous version. Maybe I will take a try this weekend. If so I will let you know. Thank you for your reply.

[Bryan Turner]

Try setting `process.nio.enabled=false` in `bitbucket.properties`, restart Bitbucket Server and test again.

If that addresses the problem, can you please update and confirm for me that it worked?

Best regards,
Bryan Turner
Atlassian Bitbucket

[mans]

I've made the change suggested and will run with this server. Since I can manually cause the issue, it only happens due to unknown reasons, I cannot say whether it for sure helps. But I will keep it running for another week or two and if it doesn't show up then it must be working.

[mans]

Suggested workaround did not help. Error occured again within 30min.

[Bryan Turner]

@mans,

It's probably worth creating a support ticket so we can work with you more directly to gather information and troubleshoot the issue.

Do you know what version of Git is installed on the server, and on the client(s) making requests that fail? (Nevermind the SSL question; you already answered it above.)

Best regards,
Bryan Turner
Atlassian Bitbucket

[mans]

Support ticket created: CA-969276

[mans]

Git version on server is 2.26.0

Unsure about the client as I haven't been able to identify which client is causing the issue.

[mans]

Git version on clients that has caused the problem varies but all of them seems to be running Apple Git up until now. Two of the versions listed below.

2.20.1
2.24.1

[mans]

I see similar behavior with git-http-backend running continuously and taking up all the CPU resources on the server. Running 7.1.0 on Linux.

[Bryan Turner]

@mans,

Can you confirm whether your Bitbucket Server is configured with SSL or without? (Having an HTTPS URL where SSL is terminated at a load balancer doesn't count; I'm specifically interested in whether you're using an SSL connector configuration on Bitbucket Server/Tomcat.)

Best regards,
Bryan Turner
Atlassian Bitbucket

[mans]

Yes, I'm running with TLS configured on the server.