Bitbucket Runners Autoscaler (k8s): DIND Authentication Failure with Private Docker Registry
Hola Atlassian Community,
I'm using Bitbucket Runners Autoscaler on a Kubernetes cluster (currently single-node). The autoscaler itself is working as expected, launching new runners pods when pipelines are triggered.
The problem:
The issue arises when a pipeline step needs to pull a Docker image from our private registry, hosted a ex private-registry.domain.org. The Docker-in-Docker (DIND) service within the runner pod fails to authenticate with this registry and cannot pull the required image.
Verification steps:
1. Successful manual test: I've confirmed that authentication works if I intervene manually. By accessing a running runner pod (using kubectl exec) and manually creating the ~/.docker/config.json. The Docker-in-Docker (DIND) service within the runner pod fails to authenticate with this registry and cannot pull the required image.
{
"auths": {
"private-registry.domain.org": {
"auth": "bXl1c2VyOm15cGFzc3dvcmQ=" // (Base64 encoded credentials)
}
}
}from within that same container, I can then successfully run docker pull private-registry.domain.org/image:tag without any authentication issues.
2. Failure During Automated Execution (DIND): However, when the pipeline runs automatically via the autoscaler-managed runner, the DIND service's attempt to pull the image from the private registry with the following error:
Unable to pull image: Head "https://private-registry.domain.org/v2/alpine/manifests/latest": no basic auth credentials
Refer to the Troubleshooting runners document for help resolving the issue.
Partial Conclusion:
This indicates to me that while the credentials themselves are valid, they are not being made accessible to, or used by, the DIND environment that the runner spins up to execute pipeline steps requiring Docker images. The problem seems specific to getting the credentials to the DIND service itself, not necessarily to the main runner container.
Question:
What is the correct and recommended method for configuring the Bitbucket Runners Autoscaler (potentially via its Helm chart or specific settings) so that private Docker registry credentials are made available to, and automatically used by, the DIND service within the generated runner pods?
Should a Kubernetes Secret be mounted to a specific path within the DIND container? Are there specific environment variables that the DIND service or the runner's startup script expects?
I would appreciate any help or guidance on how to correctly configure DIND authentication for private registries in this setup.
EDIT:
Following the kustomize logic, I mounted a volume in /root/.docker from the cm-job-template.yaml.
https://support.atlassian.com/bitbucket-cloud/docs/use-your-docker-images-in-self-hosted-runners/
Now the error is returned by the registry as unauthorized:
Unable to pull image: unauthorized:
<html>\r\n<head><title>401 Authorization Required</title></head>\r\n<body>\r\n<center><h1>401 Authorization Required</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n
EDIT: Solution Implemented & Remaining Limitation
Thank you all for the input. I wanted to share the solution I eventually implemented and a minor limitation I encountered.
Initial Workaround (Abandoned):
Initially, I managed to work around the direct authentication issue by routing registry access through a proxy server that used IP whitelisting for authorization. While this technically allowed access, it introduced significant delays, making image download times during pipeline execution unacceptably long.
Final Solution (Improved Performance):
Due to the performance issues with the proxy, I opted for a different strategy focused on local access speed:
- Local Unsecured Registry: Deployed a standard private Docker registry container directly on the same Docker daemon that hosts the Kubernetes (k8s) node. This registry was configured to run without authentication (unsecured) for simplicity and speed, relying on network isolation instead.
- Customized DIND Image: Created a customized
docker:dind(Docker-in-Docker) image. The key modification was configuring the Docker daemon inside this DIND image to trust and allow pulling from the local, unsecured registry. This typically involves adding the--insecure-registry <registry-host-ip>:<port>flag to the DIND daemon's startup parameters. - Autoscaler Deployment: Configured the Bitbucket Runners Autoscaler (using its Helm chart/settings) to utilize this customized
docker:dindimage for the runners it provisions.
This setup allows the DIND service within each runner pod to pull images very quickly from the registry located on the same host machine.
Remaining Limitation (Hostname Resolution):
The only drawback with this approach is accessing the local registry from within the DIND container using a hostname. Modifying the /etc/hosts file inside the docker:dind container (either during build or runtime) proved problematic. Therefore, any references to the private registry within pipeline steps (e.g., in image: definitions or docker pull commands inside scripts) must use the K8s node's IP address and the registry's port (e.g., image: 192.168.1.100:5000/my-image:latest) instead of a hostname like my-local-registry:5000.
Architecture Diagram:
Best Regards,
Felipe Rodriguez
