New Feature: Step Failure Strategies

[#1]

Hi all - We're excited to announce a new capability in Bitbucket Pipelines we're calling Step Failure Strategies.

Detailed information is available on our announcement blog: 

https://www.atlassian.com/blog/bitbucket/introducing-step-failure-strategies-in-bitbucket-pipelines

[Ivan Peevski]

Additional suggestions for strategies

- rerun on a different runner (4x, legacy, on-prem)

- rerun on a different docker image (fallback to an older runner)

- notify - same as ignore, but also send email/slack message to someone to highlight the issue

- custom action(s) - list of steps to be executed on failure (a bit like how `after-script` runs). The above (notify) can be done using this instead of it being its own strategy. This can also allow for reports to be added to the PR

[Jonathan Hein]

An end early and mark pipeline complete (but not failed) option would be a nice Step Failure Strategy.

Example: When it's detected the test suite has run successfully previously for this changeset.

[Michal Šimon]

It would also be cool to set this on an external commit status via the API. SaaS Tools like static code analyzers can report back to Bitbucket in a more granular way and help control the flow.

[Edmund Munday]

Hey @Michal Šimon - that's really interesting you mention that, this is a concept we've been playing with in a different way via a concept we're calling "remote steps".

Effectively a remote step would work by defining a URL that we would effectively fire a webhook event at, with the payload including all the context about the build + a short-lived authentication token that could be used to call back to the Bitbucket API.

This token would be strictly scoped down to only allow operations against the specific Pipeline and Step that it was generated for (basically it's just for updating the status of the step itself).

The remote step would be in a "pending" status until the target system calls back to Bitbucket and updates the status of the step to some kind of "terminal state" (e.g. success, fail, etc).

You could then run some kind of 3rd party/external workflow outside of Pipelines, and then call back when a final result has been achieved to update the status of the step.

The big benefit of this model is that it means you aren't burning any build minutes while you're waiting for the external system to run. With current implementations, you'd effectively have to keep the step alive the whole time, and poll the external tool instead.

---

Is this kind of concept what you're thinking?

Would be great to get your thoughts on the idea, it's not something we're actively building now, but it's something we are wanting to look at in the near future.

[Michal Šimon]

Hey Edmund, thank you for the answer and detailed description. :) This is very interesting. We are operating such a SaaS service and would love to integrate deeper if possible. With some of our customers, we implemented exactly what you described and it is suboptimal to artificially wait for an external service. A webhook URL for updating the pipeline step would be awesome.

The authentication seems reasonable, however, since we are calling many other bitbucket APIs with the standard authentication token, it would be a bit impractical to use a different token to update a build status. I would personally prefer sticking with the standard token. The short-lived token sounds to me like a great addition that can be used for 3rd parties that need only this.

If I may suggest one more thing: In the step creation API call I would love to see some sort of a default behavior configuration. For example: If the 3rd party doesn't respond in the next 15 minutes, set the step as success/fail/timed-out. This would prevent (in case of a failure at the 3rd party service) an awkward state of infinite external steps. Also, providing a link to report why the step failed would be nice.

Anyway, good examples would be:

• static code analysis with some thresholds - if the code quality is too bad, we fail the pipeline.
• code coverage with thresholds - Bitbucket pipeline runs the tests and uploads results to 3rd party for visualization. If the coverage drops too much, we fail the pipeline.
• dependencies scanning - if a serious vulnerability is found in some library, we fail the pipeline

Covering such scenarios would significantly improve the experience for many customers. If there is anything I can help with, let me know.

[Saman Jalilian]

we started testing this "ignore" strategy, it's a cool version, it would be good if you consider my points as well. 

1 - Being able to change the status of the whole pipeline in case of a failed step. Currently, since it ignores, pipelines's status would be always "success". we implemented a logic to check each steps status and create a step to check and make the pipeline status changes, it was good but we faced another issue. explained in second point.

2 - we're using triggering pipes, and using "MAX_RETRIES_ON_FAILURE" param. Noticed that re-runs are not working for the steps configured with "ignore" which is not our desire and it's better to have this.

[Matt Hudson]

Is it possible to get the on fail step status in the next step?

[Paula Reponen]

I would propose to add a failure strategy something like "continue". This would

• continue to the next steps even if a step fails
• mark the whole pipeline as failed if some steps failed
• allow to run the failed steps with 'Rerun failed steps'

This would be useful for example when tests are divided into several steps, and they can't be run in parallel, and we want to run the whole test set even if there are some failures.

Is there something like this planned?

[Niko Dojan]

Hi. Cool feature, really useful.

But please, please allow on-fail also on deployment steps.