Step in pipeline should always run, but it doesn't do that

[Bart Van Raemdonck]

I want to make it so that the step "merge-report" always runs even if one or all shards fail. But this step is not run if one step or all steps fail.

This is my yml file:

image: mcr.microsoft.com/playwright:v1.50.1-noble

definitions:
  steps:
    - step: &install-playwright
        name: Install Playwright
        script:
          - npm config set registry "https://artifactory.persgroep.cloud/artifactory/api/npm/npm"
          - curl -u${ARTIFACTORY_USER}:${ARTIFACTORY_PASSWORD} "https://artifactory.persgroep.cloud/artifactory/api/npm/auth" > .npmrc
          - npm config fix
          - npm ci
          - npm install -D @playwright/test@latest
          - npx playwright --version
          - npx envinfo --preset playwright
          - PLAYWRIGHT_BROWSERS_PATH=./ms-playwright npx playwright install --with-deps
          - PLAYWRIGHT_BROWSERS_PATH=./ms-playwright npx playwright install chrome
          - PLAYWRIGHT_BROWSERS_PATH=./ms-playwright npx playwright install msedge
          - PLAYWRIGHT_BROWSERS_PATH=./ms-playwright npx playwright install firefox
        artifacts:
          - ms-playwright/**
          - node_modules/**
        caches:
          - node

    - step: &playwright-ui-acc-tests
        name: Run Playwright UI tests
        script:
          - npx playwright --version
          - npx envinfo --preset playwright
          - npx playwright install chrome
          - npx playwright install msedge
          - export SHARD_INDEX=$(expr ${BITBUCKET_PARALLEL_STEP} + 1)
          - mkdir -p blob-report  # Collect all reports here
          - PLAYWRIGHT_BROWSERS_PATH=./ms-playwright npx playwright test --grep-invert @prod --shard=${SHARD_INDEX}/${BITBUCKET_PARALLEL_STEP_COUNT} --output=blob-report-${SHARD_INDEX}
          - echo "Playwright tests on shard ${SHARD_INDEX} completed."
          - ls -lahR blob-report/blob-report-${SHARD_INDEX} || echo "No reports found!"
          - set +e echo "This step will run and might fail"
        artifacts:
          - blob-report/**
        allow_failure: true

    - step: &playwright-ui-prod-tests
        name: Run Playwright UI tests
        script:
          - npx playwright --version
          - npx envinfo --preset playwright
          - npx playwright install chrome
          - npx playwright install msedge
          - export SHARD_INDEX=$(expr ${BITBUCKET_PARALLEL_STEP} + 1)
          - mkdir -p blob-report  # Collect all reports here
          - PLAYWRIGHT_BROWSERS_PATH=./ms-playwright npx playwright test --grep @prod --shard=${SHARD_INDEX}/${BITBUCKET_PARALLEL_STEP_COUNT} --output=blob-report-${SHARD_INDEX}
          - echo "Playwright tests on shard ${SHARD_INDEX} completed."
          - ls -lahR blob-report/blob-report-${SHARD_INDEX} || echo "No reports found!"
          - set +e echo "This step will run and might fail"
        artifacts:
          - blob-report/**
        allow_failure: true

    - step: &playwright-ui-all-tests
        name: Run Playwright UI tests
        script:
          - npx playwright --version
          - npx envinfo --preset playwright
          - npx playwright install chrome
          - npx playwright install msedge
          - export SHARD_INDEX=$(expr ${BITBUCKET_PARALLEL_STEP} + 1)
          - mkdir -p blob-report  # Collect all reports here
          - PLAYWRIGHT_BROWSERS_PATH=./ms-playwright npx playwright test --shard=${SHARD_INDEX}/${BITBUCKET_PARALLEL_STEP_COUNT} --output=blob-report-${SHARD_INDEX}
          - echo "Playwright tests on shard ${SHARD_INDEX} completed."
          - ls -lahR blob-report/blob-report-${SHARD_INDEX} || echo "No reports found!"
          - set +e echo "This step will run and might fail"
        artifacts:
          - blob-report/**
        allow_failure: true

    - step: &merge-reports
        name: Merge Playwright Reports
        script:
          - set +e echo "This step should always run"
          - npx playwright merge-reports blob-report --reporter=html
        artifacts:
          - playwright-report/**
        when: always

pipelines:
  custom:
    run-all-tests-parallel:
      - step: *install-playwright
      - parallel:
          - step:
              <<: *playwright-ui-all-tests
              name: Run PW UI tests 1/4
          - step:
              <<: *playwright-ui-all-tests
              name: Run PW UI tests 2/4
          - step:
              <<: *playwright-ui-all-tests
              name: Run PW UI tests 3/4
          - step:
              <<: *playwright-ui-all-tests
              name: Run PW UI tests 4/4
      - step:
          <<: *merge-reports
          name: Merge Playwright Reports
          when: always

    run-acceptance-tests-parallel:
      - step: *install-playwright
      - parallel:
          - step:
              <<: *playwright-ui-acc-tests
              name: Run PW UI tests 1/4
          - step:
              <<: *playwright-ui-acc-tests
              name: Run PW UI tests 2/4
          - step:
              <<: *playwright-ui-acc-tests
              name: Run PW UI tests 3/4
          - step:
              <<: *playwright-ui-acc-tests
              name: Run PW UI tests 4/4
      - step:
          <<: *merge-reports
          name: Merge Playwright Reports
          when: always

    run-production-tests-parallel:
      - step: *install-playwright
      - parallel:
          - step:
              <<: *playwright-ui-prod-tests
              name: Run PW UI tests 1/4
          - step:
              <<: *playwright-ui-prod-tests
              name: Run PW UI tests 2/4
          - step:
              <<: *playwright-ui-prod-tests
              name: Run PW UI tests 3/4
          - step:
              <<: *playwright-ui-prod-tests
              name: Run PW UI tests 4/4
      - step:
          <<: *merge-reports
          name: Merge Playwright Reports
          when: always

[Bart Van Raemdonck · Accepted solution]

If found the solution:

on-fail:
  strategy: ignore