Problem with git-upload-pack

[shay haper]

We are seeing "git fsck" return hundreds of missing or corrupt trees in some of our repositories. This causes users to see errors when viewing Pull Requests in the Bitbucket UI. We are running a 3 node Bitbucket Data Center cluster on RedHat 7 VMs using TCP/IP clustering (not multicast). The Servers are configured with one NIC which serves both the inter-cluster communication and user requests. Most of the objects returned by git fsck exist on disk in ./objects but have 0KB size. Looking in our backups, we do not see these missing trees in a valid state which makes us think that they are being created as 0 byte objects. git gc --prune=all does not help. Also, we do not find these trees in an exhaustive search of the rev-list, using ls-tree across every entry. Do you know why we might have so many 0 byte objects being created in our repos? It is a serious problem which blocks our uses from merging pull requests, affecting their daily work.

[Mark A_]

Hey Shay,

Is this issue still active for you?

Can you detail the actual issue or error you are receiving from git-upload-pack that prompted your investigation?

Do you know what activity transpired to cause the corruption? Was there an unexpected power outage or disk failure?

It is very interesting that objects are being written with 0kb as the objects in this folder are your commits. Have you experienced any other issues outside of viewing Pull Requests?

We do not recommend running a prune all as this action can delete necessary objects that are being used in forked repositories. Removing them, therefore, breaks the forked repository as the referenced objects will no longer be available. 

In the event your repositories are corrupted, we have a KB detailing the recommended action plan for when a repository becomes corrupted.