Connect to service container from a Pipe

[moritz_matchory_com]

TL;DR: I cannot reach a service container from a custom Pipe. Is this by design, and what can I do about it?

---

We're using Tailscale to connect to our internal services. Usually, this doesn't really affect Bitbucket deployments, but recently, we started tracking dependencies and vulnerabilities using Dependency Track, which is hosted in our Tailnet exclusively—so, not reachable from the outside.

Essentially, this means we want to collect information from a built Docker image as an SBOM, and send that in an API request to our Dependency Track instance. I actually created a new (public) Pipe for this: matchory/dependency-track-pipe. It can be used like so:

- pipe: matchory/dependency-track-pipe:0.4.2
  variables:
    SERVER_URL: "https://dtrack.example.com"
    API_KEY: $DTRACK_API_KEY

Since we cannot connect directly, we need to run the Tailscale client in the pipeline, and use its SOCKS 5 proxy to forward requests from our pipeline to the server.
The easiest way to do this is to run a Tailscale service container:

definitions:
  services:
    tailscale:
      image: tailscale/tailscale:latest
      variables:
        KUBERNETES_SERVICE_HOST: ""
        TS_KUBE_SECRET: ""
        TS_ACCEPT_DNS: "true"
        TS_AUTHKEY: $TAILSCALE_AUTHKEY
        TS_SOCKS5_SERVER: "localhost:1055"

(If you're also trying to get Tailscale to work, note the Kubernetes variables. These are required to avoid Tailscale detecting the Bitbucket k8s network, which breaks connectivity for your pipeline.)

This allows connecting to all Tailnet services via the SOCKS5 proxy at socks5://localhost:1055. You can verify this by passing the proxy address to curl, for example: 

ALL_PROXY=socks5://localhost:1055 \
  curl https://some-internal-service.your-tailnet.ts.net/

Plugging all of this together, we want to

1. set up a proxy into our Tailscale network,
2. collect an SBOM from the container we've just built, and
3. send that SBOM to our Dependency Track instance via the proxy.

The following pipeline should be able to achieve this:

definitions:
  services:
    tailscale:
      image: tailscale/tailscale:latest
      variables:
        KUBERNETES_SERVICE_HOST: ""
        TS_KUBE_SECRET: ""
        TS_ACCEPT_DNS: "true"
        TS_AUTHKEY: $TAILSCALE_AUTHKEY
        TS_SOCKS5_SERVER: "localhost:1055"

  build-and-push: &build-and-push
    # ...

pipelines:
  branches:
    main:
      - step:
          name: Build and push
          services:
            - docker
          script:
            - *build-and-push
      - step:
          name: Collect SBOM
          services:
            - docker
            - tailscale
          script:
            - pipe: matchory/dependency-track-pipe:0.4.2
              variables:
                ALL_PROXY: socks5://localhost:1055
                SERVER_URL: https://dtrack.your-tailnet.ts.net/
                # ...

This does not work, however, because the proxy cannot be reached from within the pipe container:

SBOM Upload failed TypeError: fetch failed
    at fetch (/pipe/node_modules/undici/index.js:112:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async main (file:///pipe/upload.mjs:80:22) {
  [cause]: SocksClientError: connect ECONNREFUSED 127.0.0.1:1055
      at SocksClient.closeSocket (/pipe/node_modules/socks/build/client/socksclient.js:390:32)
      at SocksClient.onErrorHandler (/pipe/node_modules/socks/build/client/socksclient.js:363:14)
      at Socket.onError (/pipe/node_modules/socks/build/client/socksclient.js:225:38)
      at Object.onceWrapper (node:events:634:26)
      at Socket.emit (node:events:519:28)
      at emitErrorNT (node:internal/streams/destroy:170:8)
      at emitErrorCloseNT (node:internal/streams/destroy:129:3)
      at process.processTicksAndRejections (node:internal/process/task_queues:90:21) {
    options: {
      command: 'connect',
      proxy: [Object],
      timeout: 10000,
      destination: [Object],
      existing_socket: undefined
    }
  }
}

I think this may be related to the way pipe networking is set up, but there is nothing in the documentation mentioning limitations in this regard, neither in "Databases and Service containers", nor in "Use Pipes in Bitbucket Pipelines".

I could include Tailscale in the Dependency Track pipe container, but that would make it way too specialised for everyone else to use, and I think that'd be a sad state of affairs for Pipelines.

[Patrik S]

Hey @moritz_matchory_com ,

thanks for reaching out!

Pipes containers in a build essentially run in the docker service.

When you try to access another service using the localhost IP (127.0.0.1) from within a pipe, that IP is actually referred to within the step's docker service scope, so it won't be able to connect.

We have the following article that cover a sscenario on how to connect to a service container while inside a Docker container on Bitbucket Cloud Pipelines : 

• https://confluence.atlassian.com/bbkb/connecting-to-a-service-container-while-inside-a-docker-container-on-bitbucket-cloud-pipelines-1217200735.html

That article can be expanded to pipes as well, if you take into consideration that a pipe is a docker container spun up in the docker service.

So the same solution proposed in that article should be applicable to your use case.

The pipe start command already includes the --add-host argument by default:

--add-host="host.docker.internal:$BITBUCKET_DOCKER_HOST_INTERNAL"

which will map the host.docker.internal to the build container's network.

You should then be able to use the host.docker.internal address to access your service container that has the socks 5 proxy.

That being said, could you try replacing the IP/port from 127.0.0.1:1055 to host.docker.internal:1055 and let us know how it goes?

Thank you, @moritz_matchory_com !

Patrik S

[moritz_matchory_com]

Hey Patrik,

that's interesting, the page on variables and secrets doesn't include BITBUCKET_DOCKER_HOST_INTERNAL. However, I tried it, and while host.docker.internal indeed seems to resolve to another host, it still refuses the connection:

ECONNREFUSED 10.39.25.100:1055

That's weird.

[Igor Stoyanov]

@moritz_matchory_com hi.
Try to connect to your service container without pipe usage, just on pipelines level to determine where is the problem. Maybe the problem is with your configuration.

Regards, Igor

[moritz_matchory_com]

I did validate the Tailscale container by connecting via the proxy from the build step directly as opposed to the pipe.
That is, this works:

- curl --proxy sock5h://localhost:1055

But this does not:

- pipe: matchory/dependency-track-pipe
  variables:
    ALL_PROXY: "sock5h://localhost:1055"

And neither does this:

- pipe: matchory/dependency-track-pipe
  variables:
    ALL_PROXY: "sock5h://host.docker.local:1055"

[Igor Stoyanov]

Hi @moritz_matchory_com . Try to add DOCKER_HOST to your pipe variables: example.

Regards, Igor