Triggering Multiple Pipelines for One Repository

[Jonny Carter]

I have a repository that builds and deploys several submodules. These submodules share some common code defined in shared modules within the repository.

I'm trying to un-complicate the current setup, which has multiple other repositories with distinct pipeline definitions that manually clone the source repo and run a determined series of build steps.

The way we currently trigger the other pipelines is using the trigger pipeline pipe (source at https://bitbucket.org/atlassian/trigger-pipeline/src/master/).

I can use the same pipe to trigger pipelines within the same repository. That helps some, because I can more easily reuse pipeline steps using YAML anchors and it shaves some time off the build process.

The trouble I have is that if I kick off more than two pipelines, only two will show up on the pull request, even if they are all pipelines within the same repository on the same branch. For pull requests that cause any of the submodule builds to fail, I definitely don't want the PR to be mergeable.

I can tell the trigger-pipeline pipe to wait for the result from the other pipeline, but this effectively doubles the amount of build minutes I pay for, which is needlessly expensive.

I have searched, but can't find out if this is a documented limitation of Bitbucket Pipelines or just a quirk of the UI. Is there some setting somewhere that I can use to allow more than 2 pipeline results to be associated with a pull request? I have tried passing the BITBUCKET_PR_ID, BITBUCKET_PR_DESTINATION_BRANCH, BRANCH_NAME, and other relevant variables to the child pipeline using the `PIPELINE_VARIABLES` variable defined by trigger-pipeline, but I still only get the top two hits.

Is there some more scalable way to associate lots of pipelines back to one pull request than the trigger-pipeline pipe? I'd happily put them all in nested `parallel` blocks, but that isn't really an option in Bitbucket Pipelines.

[Theodora Boudale]

Hi Jonny,

The builds that are displayed in the PR depend on the build statuses associated with the last commit of the PR's source branch. There is no hard limit of 2 build statuses, it can be just one, it can also be multiple; it depends on what pipeline definitions are running and also how the builds get triggered.

That being said, pipelines can update the existing build status of a commit instead of creating a new one (e.g. this happens with custom builds, see feature request: https://jira.atlassian.com/browse/BCLOUD-23256). I went ahead and created a feature request for creating a separate build status for each pipeline that runs on a branch/commit:

• https://jira.atlassian.com/browse/BCLOUD-23605

You can add your vote to that feature request (by selecting the link Vote for this issue) to increase the chances of this being implemented. You are more than welcome to leave feedback, and you can also add yourself as a watcher (by selecting the link Start watching this issue) if you'd like to be notified via email on updates.

You also raised another issue:

For pull requests that cause any of the submodule builds to fail, I definitely don't want the PR to be mergeable.

I can tell the trigger-pipeline pipe to wait for the result from the other pipeline, but this effectively doubles the amount of build minutes I pay for, which is needlessly expensive.

I assume that the submodule pipelines are defined in in the bitbucket-pipelines.yml of each submodule repo, and so they run on the submodule repo? If so, and if you'd like to keep this setup, then using the WAIT variable is the only way.

However, you could work around this the following way:

Instead of running the submodule builds in the submodule repos, you could run them in the main repo. You could define a custom pipeline for each submodule. In each custom pipeline, you could clone the respective submodule and then run your tests there. You could also disable cloning of the main repo for these pipelines, if it's not needed for the submodule builds.

We have the following blog post that explains how to clone another repo in pipelines (you'll need to set up authentication):

• https://www.atlassian.com/blog/bitbucket/cloning-another-bitbucket-repository-in-bitbucket-pipelines

One thing that is missing from this article is that an SSH client needs to be installed in the Docker image you use a build container, so you can clone via SSH.

Please feel free to reach out if you have any questions.

Kind regards,
Theodora

[Jonny Carter]

Thanks for the reply, @Theodora! 

> I assume that the submodule pipelines are defined in in the bitbucket-pipelines.yml of each submodule repo, and so they run on the submodule repo? If so, and if you'd like to keep this setup, then using the WAIT variable is the only way.

I should clarify: the source code for each submodule is in the same repository. The pipeline definition is currently in another repository, but I want to move it so that the pipeline definition lives alongside the code. In part, this is to avoid re-cloning one repository from another pipeline. That adds lag and complexity to the pipelines, and I'm working to simplify that.

Let me share a screenshot to help explain. When I open a PR in the repository in question, the `pull-requests: **` pipeline kicks off. It looks like this.

There are three steps in this pipeline that use the aforementioned trigger-pipeline pipe to kick off three other custom pipelines in the same repository.

However, the pipelines card on the right hand side of the pull request looks like this:

2/2 builds, even though there are at least 4 distinct pipelines. The root one (which is 9037 in the above screenshot) and three "child" pipelines spawned using the trigger-pipeline pipe.

The feature requests you raised are definitely related, but it's not clear to me that they address the felt need here. These are 4 distinct pipelines with different identifiers. They are building against the same branch. Is there some metadata that I can add to these pipelines to make them associated with the same pull request so that they:

1. Show up in the UI
2. Block merge if one of them fails

[Syahrul]

Hey @Jonny Carter 

How is the second pipeline, pipeline #9038, triggered? Was it initiated by one of the steps in build #9037? I need to fully understand the situation and find a way to reproduce it. Could you please raise a support ticket on our Support Portal and include a link to this community post as this will help expedite the investigation.

Regards,
Syahrul 

[Aron Gombas _Midori_]

@Jonny Carter I don't have a real answer, but maybe two hints to investigate further:

1. Dynamic Pipelines is a mechanism to better orchestrate your builds if you don't mind to develop a small-ish Forge app for that purpose. It may worth the effort!
2. Pipeline Forms is a Bitbucket Cloud app to trigger builds with user-picked parameters, periodically, etc. It doesn't require programming skills.

Maybe one of these will be helpful.

[Jonny Carter]

I can imagine that Dynamic Pipelines could do it. I haven't been willing to futz with it because I'm categorically against buying in too hard to any particular CI system. My view is CI systems should be commodity products with very low vendor lock-in. Dynamic Pipelines smell like more commitment and maintenance burden than I really wanna take on for this need.

It also seems like the kind of workflow Bitbucket should be able to support out of the box. All the competitors (GitLab, GitHub, heck, even Bitbucket DC which has no built-in CI) can support showing more than two CI builds on a PR.