Bitbucket Pipe Trigger Pipeline hangs

[zoltan_zvara]

The following pipe hangs at the end of the otherwise working pipeline.

- pipe: atlassian/trigger-pipeline:5.0.1
  variables:
    BITBUCKET_USERNAME: $BITBUCKET_USERNAME
    BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
    REPOSITORY: 'repository-name'
    DEBUG: 'true'

The full `bitbucket-pipelines.yml` are as follows:

image:
  name: $REDACTED/development/build-default:latest
  username: $REGISTRY_USERNAME
  password: $REGISTRY_PASSWORD

pipelines:
  default:
    - step:
        # @note Please note that the build process is grouped into a single "super step",
        #       because Bitbucket Pipelines does not cache the above private build image
        #       for any step. Before any step, it is pulled again from the registry, thus,
        #       rendering each step slow by adding a large overhead of a Docker image pull.
        # @see [https://jira.atlassian.com/browse/BCLOUD-19918]
        # Once the above issue `BCLOUD-19918` is fixed, this Pipeline then shall be split
        # into many and parallel steps to better load-balance build across Kubernetes clusters.
        name: Super Step
        runs-on:
          - self.hosted
          - linux
        script:
          - cp $REDACTED/NuGet.config ~/.nuget/NuGet/NuGet.Config
          - |
            dotnet nuget update source $SONATYPE_HOST \
            --username $SONATYPE_USERNAME --password $SONATYPE_PASSWORD \
            --store-password-in-clear-text
          - git config user.name $GIT_USERNAME
          - git config user.email $GIT_MAIL
          - dotnet tool restore --no-cache --configfile ~/.nuget/NuGet/NuGet.Config
          - dotnet restore --no-cache --configfile ~/.nuget/NuGet/NuGet.Config
          - dotnet build --no-cache --no-restore
          - git diff --quiet HEAD || git commit -am "Code quality changes. [skip ci]"
          - git push
          - dotnet test --no-build
          - dotnet pack src/Common --output nugets --no-build
          - dotnet pack src/Generator --output nugets --no-build
          - dotnet pack src/Test/Common --output nugets --no-build
          - dotnet nuget push nugets/*.nupkg -k $SONATYPE_API_KEY -s https://$SONATYPE_HOST/repository/nuget-releases/
          - bump2version patch
          - git push
          - pipe: atlassian/trigger-pipeline:5.0.1
            variables:
              BITBUCKET_USERNAME: $BITBUCKET_USERNAME
              BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
              REPOSITORY: 'repository-name'
              DEBUG: 'true'

Logs are as follows (please note that $REGISTRY_USERNAME is substituted to "bitbucket"):

Unable to find image '$REGISTRY_USERNAMEpipelines/trigger-pipeline:5.0.1' locally
5.0.1: Pulling from $REGISTRY_USERNAMEpipelines/trigger-pipeline
461246efe0a7: Pulling fs layer
e37ebf440f7f: Pulling fs layer
ceeacead7688: Pulling fs layer
975f1970522b: Pulling fs layer
824120c69d4c: Pulling fs layer
ef5c4a105c3c: Pulling fs layer
94c103dd148e: Pulling fs layer
922ee4c5c8c5: Pulling fs layer
959a68abbb57: Pulling fs layer
975f1970522b: Waiting
922ee4c5c8c5: Waiting
94c103dd148e: Waiting
ef5c4a105c3c: Waiting
959a68abbb57: Waiting
e37ebf440f7f: Verifying Checksum
e37ebf440f7f: Download complete
ceeacead7688: Verifying Checksum
ceeacead7688: Download complete
975f1970522b: Verifying Checksum
975f1970522b: Download complete
461246efe0a7: Verifying Checksum
461246efe0a7: Download complete
824120c69d4c: Verifying Checksum
824120c69d4c: Download complete
ef5c4a105c3c: Verifying Checksum
ef5c4a105c3c: Download complete
94c103dd148e: Verifying Checksum
94c103dd148e: Download complete
922ee4c5c8c5: Verifying Checksum
922ee4c5c8c5: Download complete
959a68abbb57: Verifying Checksum
959a68abbb57: Download complete
461246efe0a7: Pull complete
e37ebf440f7f: Pull complete
ceeacead7688: Pull complete
975f1970522b: Pull complete
824120c69d4c: Pull complete
ef5c4a105c3c: Pull complete
94c103dd148e: Pull complete
922ee4c5c8c5: Pull complete
959a68abbb57: Pull complete
Digest: sha256:f6effedb090ab2c1e936b7b382dd4512b810a52033706459095e6089fc7e19c7
Status: Downloaded newer image for $REGISTRY_USERNAMEpipelines/trigger-pipeline:5.0.1
DEBUG: Starting new HTTPS connection (1): $REGISTRY_USERNAME.org:443
DEBUG: https://$REGISTRY_USERNAME.org:443 "GET /$REGISTRY_USERNAMEpipelines/official-pipes/raw/master/pipes.prod.json HTTP/1.1" 200 None
DEBUG: Starting new HTTPS connection (1): api.$REGISTRY_USERNAME.org:443

The `$BITBUCKET_USERNAME` has Admin access to the repository running this pipeline and Admin access to the repository for which the Pipeline should be triggered.

The $REPOSITORY has one branch, "master," and is in the same namespace as the current pipeline.

`docker ps` shows the Runner that the Docker container is not running that should be running the "trigger-pipeline" pipe itself.

[zoltan_zvara · Accepted solution]

As @Oleksandr Kyrdan suggested, it works in a separate step but not as part of a script in the previous step.

[Patrik S]

Hello @zoltan_zvara ,

Thank you for reaching out to Atlassian Community!

In order for us to investigate further, could you please share the full YML file you are using in this repository? 

Also, are the pipe logs you attached the full logs? I'm asking because the end of the logs seems to be truncated, so I'm wondering if it's because the pipe itself hangs or if you maybe have not shared the full logs. If the later, could you please share the full logs generated by the pipe so we can analyze them? 

Thank you, @zoltan_zvara !

Patrik S

[zoltan_zvara]

Hi @Patrik S

Thanks for helping me with this issue.

I improved the initial post to include the entire pipeline (with a few redacted parts, hopefully, unrelated to the issue).

I confirmed that the pipe logs are as complete as possible by:

- waiting for the pipeline to get stuck and then copying the pipe logs from the end;
- while the "$REDACTED/development/build-default:latest" is running, I "tailed" it to see the logs from the Docker container.

In both cases, these are the actual logs extracted from the pipe, nothing more. I also noticed that the Docker container running the pipe has exited and has been deleted from the Docker engine (I could not recover its original logs or its exit state). I suspect that the container has exited unexpectedly.

I have the following idea to debug the issue, but it could be time-consuming:

- monitor the Docker engine and attempt to collect the original pipe container's logs in some way;
- start the pipe container so that it is not removed (not sure if a pipe can be started that way?);
- attempt to run the same pipe in the Cloud to see whether the problem is related to a Docker engine crashing the pipe container.

Do you have any other ideas or tips? Did you observe a similar issue with Bitbucket Runners or Cloud pipelines?

[Oleksandr Kyrdan]

Hi @zoltan_zvara 

Could you move trigger-pipeline pipe to the separate step?
And provide us with logs output because current logs without any errors :

          - pipe: atlassian/trigger-pipeline:5.0.1
            variables:
              BITBUCKET_USERNAME: $BITBUCKET_USERNAME
              BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
              REPOSITORY: 'repository-name'
              DEBUG: 'true'

Best regards,
Oleksandr Kyrdan

[zoltan_zvara]

@Oleksandr Kyrdan I can confirm that it works in a separate step but not in the `Super Step` as described previously.