CI Console & Pipeline timing differences

[Joey Bouten]

Hello, 

Currently we are moving from another tool to bitbucket CI. We are using self-hosted runners and are pipeline is currently up to an hour. Now we've already had done some optimization steps. However we are seeing major timing differences between the console and the pipeline view left of the console. 

Example of a step: 

Runner <1s
Build setup <1m 29s
custom grep command placeholder <1s
Build teardown <1s

But the step viewer says this has taken 2m 36s

Also for a larger step like behat tests (Docker in Docker) 

Runner <1s
Build setup 2m 43s
export DOCKER_COMPOSE_VERSION=2.19.0 <1s
curl -L https://github.com/docker/compose/releases/download/v${DOCKER_COMPOSE_VERSION}/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose 12s
chmod +x /usr/local/bin/docker-compose <1s
cp .env.REDACTED.dist .env <1s
docker login REDACTED -u REDACTED -p REDACTED 15s
make REDACTED <1s
make behat-up 3m 14s
make behat-REDACTED-prepare 40s
make behat-REDACTED > behat-REDACTED.log 2>&1 || true 10m 16s
cat behat-REDACTED.log <1s
make behat-down 7s
Build teardown 51s

But the step viewer says this has taken 25m 33s


The first step is a solo sequential step, the other runs in parallel to other tests. However we see all steps run at the same time, since we have enough runners spin up. 

Is there anyway i can see what causes the time difference?

[Mark C]

Hi @Joey Bouten

Welcome to the community.

For your Pipelines YAML configuration, could you confirm if you're using a size option of 2x or higher?

Regards,
Mark C

[Joey Bouten]

Hi @Mark C 

Yes, for some steps we are using 2x or higher. 

Is this what causes the impact?

Joey

[Mark C]

Hi @Joey Bouten

Thanks for the confirmation.

Indeed, using 2x or higher will impact your build minutes consumption. As an example:

4x steps use four times the number of build minutes of 1x steps, 2x steps use twice the number of build minutes of 1x steps, and so on. Note: Increasing your build minutes may result in an increase in your current costs for pipelines builds.

Reference: https://support.atlassian.com/bitbucket-cloud/docs/global-options/#Size

Regards,
Mark C

[Joey Bouten]

Hi @Mark C 

I wasn't talking about build minutes here. But the minutes the CI takes to complete. There is a difference in what is shown left in the overview and right in the console. 

Since we are doing self-hosted we don't mind the consumption and are using 2x & 4x to reserve more power for specific steps. When hovering over the pipeline, the tooltip actually says 0 minutes consumed

Maybe this picture will clear my question up more. In most steps there is a major difference. 

[Mark C]

Hi @Joey Bouten

Thank you for that clarification, appreciate it.

I had a discussion with our internal team regarding this and this could be due to the timing when each command gets updated. For instance, if a command gave no additional output and took some time in that specific command, it may affect the total time but not the specific command time.

Additionally, we also have a public bug ticket for this and we are tracking it here: https://jira.atlassian.com/browse/BCLOUD-20979

You can watch and click the "This affects my team" button so that you'll be notified of any updates from our team when the bug is fixed.

You can also check out this relevant community post: https://community.atlassian.com/forums/Bitbucket-questions/Bitbucket-s-pipeline-timeout-doesn-t-match-the-reported-tasks/qaq-p/2393676

Regards,
Mark C