Problems upgrading git version due to broken refs

[CATS Delivery Platform]

We can't upgrade git to version 2.34 because some old repositories complain about broken references.

So, it fails with fatal error when pushing git notes refs to the server, but no errors when cloning/pushing/pulling code

remote: fatal: bad object refs/pull-requests/1/v2
fatal: bad object refs/pull-requests/1/v2
To ssh://BITBUCKET/inv/invoice-printer.git
 ! [remote rejected]   refs/notes/commits -> refs/notes/commits (missing necessary objects)
error: failed to push some refs to 'ssh://BITBUCKET/inv/invoice-printer.git'

(The actual server is REDACTED)

This is not happening with git version 2.33, and we're on a single host with Ubuntu 22. Be aware that this version of git is affected by CVE-2024-32002 and we need to upgrade ASAP.

Wrapping up our findings:

• it happens on older repos
• only affects refs/pull-requests/*
• working with the repo locally does not produce any error or warning
• git fsck/ git gc... does not complain, but it doesn't solve the problem either.
• pushing an affected repo to a new empty repo fixes the issue (but of course, we lose the PR history on the UI)
• We see inconsistencies on disk: refs/pull-requests/* vs stash-refs/pull-requests/*

As said before, it happens on some older repositories. We think these have inconsistencies between git and DB (for whatever reason), but we don't know exactly how many repositories are affected (we host thousands) and if there's a way to reconcile git and DB.

We're running out of ideas, any hint?

[CATS Delivery Platform · Accepted solution]

We just need to remove all refs/pull-requests/v*. They are not supposed to be there, and can happen when the git update is interrupted or a backup taken in the middle of the process is restored