build trigger delay (by commit)
Hello. Not a trivial problem at all, maybe someone had a similar experience or there are thoughts where to dig.
There is Bitbucekt DC and Bamboo Server - living on different nodes. Bitbucket, although DC, works in a single instance. Both are located behind nginx. Requests to nginx come from an internal corporate balancer. Presumably due to work on the balancer to enable HTTPS support (adding a new certificate and some additional settings), we noticed the following behavior:
After the commit, Bamboo processes builds with a delay of up to 1 hour, this started around the same time that the work on the balancer was done. At the same time, manual builds are launched successfully, it was also possible to add the rest api to the queue, but if you make a commit to the repo, then the Bamboo build will not be triggered. BUT if you commit not to the code but to the Bamboo specs, then they are scanned successfully and quickly.
Application links were recreated. Did restarts with clearing plugin caches. The setting on the balancer was turned off yesterday.
I would appreciate any thoughts.
Thread dump from Bamboo (bitbucket have the same situation) - many threads in waiting state
UPD: we located problem on Bamboo side by capturing package from Bitbucket very soon after commit was maded
UPD: helps increasing of threads -Dbamboo.plan.exe.threads=8 and heap memory -Xmx
