Using HTTPS, clone/fetch repositories more than 160 files fail with exit code 128 and no emessage

[Benoit van Bogaert]

Please, do not classify my question as SPAM as you just done. This problem really prevent to go on production!

After installation and using the HTTPS interface (because of internals rules, we can not use SSH), we can create a repository, clone it, add/commit files, push, pull while the repository contains a few files.

When the repository contains more files (for instance more than 160 text files, a few KB each) and when using the HTTPS url (with SSH clone/fetch work properly), the clone and fetch commands fail without any error message but exit code 128. We are using as client git version 1.8.3.1 and 1.8.2.3.

The client just says nothing (no error reported) even if we enable logs. For instance:

export GIT_TRACE_PACKET=1

export GIT_TRACE=1

export GIT_CURL_VERBOSE=1

 The server administrator can not identify the source of the problem. He confirms that we are behind a reverse proxy but the problem is also present when bypassing the reverse proxy.

For the following request, 

git -c http.sslVerify=false  clone http://<username>:<password>@webprisme-test.cfmu.eurocontrol.int/srs_bitbucket/scm/sassc/sandbox.git test

I have the following logs:

bitbucket-access.log

172.21.116.75,127.0.0.1 | https | i@YXB5R8x784x659x0 | - | 2018-11-29 13:04:30,055 | "GET /scm/sassc/sandbox.git/info/refs HTTP/1.1" | "" "git/1.8.2.3" | - | - | - | - | - | - | 172.21.116.75,127.0.0.1 | https | o@YXB5R8x784x659x0 | - | 2018-11-29 13:04:30,065 | "GET /scm/sassc/sandbox.git/info/refs HTTP/1.1" | "" "git/1.8.2.3" | 401 | 0 | 0 | - | 9 | - | 172.21.116.75,127.0.0.1 | https | i@YXB5R8x784x660x0 | - | 2018-11-29 13:04:30,070 | "GET /scm/sassc/sandbox.git/info/refs HTTP/1.1" | "" "git/1.8.2.3" | - | - | - | - | - | - | 172.21.116.75,127.0.0.1 | https | o@YXB5R8x784x660x0 | bvbogaer | 2018-11-29 13:04:30,235 | "GET /scm/sassc/sandbox.git/info/refs HTTP/1.1" | "" "git/1.8.2.3" | 200 | 0 | 380 | protocol:1, refs | 165 | - | 172.21.116.75,127.0.0.1 | https | i@YXB5R8x784x661x0 | - | 2018-11-29 13:04:30,240 | "POST /scm/sassc/sandbox.git/git-upload-pack HTTP/1.1" | "" "git/1.8.2.3" | - | - | - | - | - | - | 172.21.116.75,127.0.0.1 | https | o@YXB5R8x784x661x0 | bvbogaer | 2018-11-29 13:04:34,386 | "POST /scm/sassc/sandbox.git/git-upload-pack HTTP/1.1" | "" "git/1.8.2.3" | 200 | 192 | 1212889 | cache:bypass, clone, protocol:1 | 4146 | - |

bitbucket.log

2018-11-29 13:04:30,353 WARN [http-nio-9010-exec-4] bvbogaer @YXB5R8x784x661x0 172.21.116.75,127.0.0.1 "POST /scm/sassc/sandbox.git/git-upload-pack HTTP/1.1" c.a.util.contentcache.ContentCache 22: Caching has been temporarily disabled because there is not enough free space on /opt/webapps/shared/applications/bitbucket/srs_bitbucket/home/caches/scm/22 (1769603072 bytes free) 2018-11-29 13:04:33,006 INFO [io-pump:thread-7] bvbogaer @YXB5R8x784x661x0 172.21.116.75,127.0.0.1 "POST /scm/sassc/sandbox.git/git-upload-pack HTTP/1.1" c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection

The proxy server log shows:

172.30.173.131 - - [29/Nov/2018:13:03:53 +0000] "GET /srs_bitbucket HTTP/1.1" 302 -

172.21.116.75 - - [29/Nov/2018:13:04:30 +0000] "GET /srs_bitbucket/scm/sassc/sandbox.git/info/refs?service=git-upload-pack HTTP/1.1" 401 -

172.21.116.75 - - [29/Nov/2018:13:04:30 +0000] "GET /srs_bitbucket/scm/sassc/sandbox.git/info/refs?service=git-upload-pack HTTP/1.1" 200 380

172.21.116.75 - - [29/Nov/2018:13:04:30 +0000] "POST /srs_bitbucket/scm/sassc/sandbox.git/git-upload-pack HTTP/1.1" 200 606569

172.30.173.130 - - [29/Nov/2018:14:04:53 +0100] "GET /srs_bitbucket HTTP/1.1" 302 -

172.30.173.131 - - [29/Nov/2018:13:04:53 +0000] "GET /srs_bitbucket HTTP/1.1" 302 -

172.30.173.130 - - [29/Nov/2018:13:05:53 +0000] "GET /srs_bitbucket HTTP/1.1" 302 –

Please, note that the following workaround works because the fetch exits code 128 but leave the .git folder while clone remove the just created folder:

                git init

                git remote add origin https://www.example.com/myunit_bitbucket/myproject/myrepo/sandbox.git

                git -c http.sslVerify=false fetch origin

                git -c http.sslVerify=false pull sassc master

But this solution is not acceptable in production.

(Note: Edited by Atlassian to remove sensitive information) 

[Tomasz Tokarczuk · Accepted solution]

Hi Benoit

Sorry you've got under our spam filter aim!

We've noticed in your logs:

there is not enough free space on /opt/webapps/shared/applications/bitbucket/srs_bitbucket/home/caches/scm/22 (1769603072 bytes free)

Although it seems you have lots of space there, the default minimum value for the SCM cache is 5 GB, hence the message. To change the default value please create or edit the file BITBUCKET_HOME/bitbucket.properties to set plugin.bitbucket-scm-cache.minimum.free.space 

Could you please make sure there's enough both inodes and space on your disk?

To check if please simple issue:

For inodes:

df -i

For free space:

df -h

I would also recommend trying the same clone from the Bitbucket host itself to confirm it's not an interruption in the communication or git version issues. I would also try using a later version of Git for testing.

Best regards,
Tomasz Tokarczuk
Atlassian Support

[Benoit van Bogaert]

Thanks for answer. It works now!

[Ana Retamal]

Hi Benoit!

We're looking at your issue right now and we've noticed you had some sensitive information there which we've removed already. For security reasons, we advise you change your password as this post is public and anyone might have been able to see it. 

If you have any questions regarding that, let us know!

You'll get another response from us with more info regarding your question soon.

Kind regards,

Ana