All remaining pipeline jobs fail after period of time

[Derick Downey]

I have maybe 100 failed pipeline jobs to re-run. Each job takes about 20 minutes on our self-hosted server. (Going into each failed pipeline web page individually then clicking "re-run" takes me forever, plus I get rate-limited opening that many web pages because Atlassian thinks I'm doing a DDOS attack. It would be incredible if I was able to batch-select pipelines to re-run.) Besides that point...

Twice now, after I've attempted to re-run all of the failed pipelines, it makes its way through pipelines fine for about an hour, then suddenly all of the remaining jobs in the queue give me this error:

• Failed. Could not assign the step to a runner as all runners are busy. Consider adding more runners and/or re-run the step. Refer to the Troubleshooting runners document for help resolving the issue.

Is there a timeout, where if a pipeline is queued for too long, Bitbucket pipelines will just automatically fail it?

The runner still shows online in the Bitbucket web portal. Rerunning the `start.ps1` script on the runner does not seem to fix it. It seems I have to open PowerShell on the runner and do `Get-Process java | Stop-Process` first before running the `start.ps1` script. Then the pipelines will run fine again. If I put like 3 pipelines in the queue, it will get through all of them. If I put 50 in the queue, they will start to fail eventually. Any advice is appreciated.

[Mark C]

Hi @Derick Downey

Thank you for reaching out to the community.

For triggering Pipelines builds, you can try exploring the use of this script: https://bitbucket.org/atlassian/trigger-pipeline

For self-hosted runners, could you confirm how may self-hosted runners you're using at the moment?

Regards,
Mark C

[Derick Downey]

This pipeline example code is great! Thanks for the share.

We only have one self-hosted runner.

My current pipeline code works fine and has been working for maybe 3 months now. Every time a pull request is created or updated, it runs a "test build" on the source branch. However like I said in the original message, it takes about 20 minutes to do an entire build, although I do cache build successes so we don't build the same branch/commit combo twice. (See attached photo below.)

When there's a normal flow of pipelines coming in, everything works fine. The issue was I attended a conference and our runner fell offline, so I had about 100 jobs to run when I came back from the conference. It would run the first couple successfully, then "fail" all of the remaining ones with the note I included in the original message. Is there like a "timeout" where Bitbucket automatically cancels pipelines if they've been sitting too long?