Unable to run self hosted docker runner on GKE cluster

[Murtaza Lokhandwala]

I am trying to run a self hosted runner using Docker on GKE cluster deployment.

The following command is migrated to k8s deployment(values are dummy):

docker container run -it -v /data:/data -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/containers:/var/lib/docker/containers:ro -e ACCOUNT_UUID={87642016-abcd-xyze-b27d-1c58450bace4} -e REPOSITORY_UUID={c0c78c96-e5e7-1234-abcd-c337d17dd794} -e RUNNER_UUID={testinge-55c3-55d1-8fe5-15fbcbc06301} -e RUNTIME_PREREQUISITES_ENABLED=true -e OAUTH_CLIENT_ID=J3X0AgzCqjNLfuqeOGAOATAOVTdE7Zr -e OAUTH_CLIENT_SECRET=ATOABGamRkviXzw12345jehreHHMMc27gMGcQCGfhpOz-FJNbDb80eaRJTqjhKhHOX7Q6A2F0F185C6 -e WORKING_DIRECTORY=/data --name runner-e5e949ea-55c3-55d1-8fe5-15fbcbc06301 docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner

Following is the minimal k8s manifest

apiVersion: apps/v1
kind: Deployment
metadata:
name: bitbucket-runner
namespace: duploservices-np-apps
labels:
app: bitbucket-runner
tenantname: duploservices-np-apps
spec:
replicas: 1
selector:
matchLabels:
app: bitbucket-runner
tenantname: duploservices-np-apps
template:
metadata:
labels:
app: bitbucket-runner
tenantname: duploservices-np-apps
spec:
serviceAccountName: duploservices-np-apps-app-user
nodeSelector:
allocationtags: apps-general
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: tenantname
operator: In
values:
- duploservices-np-apps
- duploservices-npuse1
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
labelSelector:
matchLabels:
app: bitbucket-runner
topologyKey: kubernetes.io/hostname
containers:
- name: bitbucket-runner
image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:latest
imagePullPolicy: Always
envFrom:
- secretRef:
name: bitbucket-runner
env:
- name: DOCKER_IMAGE
value: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:latest
securityContext:
runAsUser: 0
volumeMounts:
- name: bitbucket-runner
mountPath: /tmp
- name: docker-containers
mountPath: /var/lib/docker/containers
- name: docker-sock
mountPath: /var/run/docker.sock
volumes:
- name: bitbucket-runner
persistentVolumeClaim:
claimName: bitbucket-runner
- name: docker-containers
hostPath:
path: /var/lib/docker/containers
type: Directory
- name: docker-sock
hostPath:
path: /var/run/docker.sock
type: Socket
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1

The machine used for deployment is Container Optimized OS machine.

NAME="Container-Optimized OS"
ID=cos
PRETTY_NAME="Container-Optimized OS from Google"
HOME_URL="https://cloud.google.com/container-optimized-os/docs"
BUG_REPORT_URL="https://cloud.google.com/container-optimized-os/docs/resources/support-policy#contact_us"
KERNEL_COMMIT_ID=e55bf22ffd8898012c8f3265949ead1ab34ed684
GOOGLE_METRICS_PRODUCT_ID=26
GOOGLE_CRASH_ID=Lakitu
VERSION=113
VERSION_ID=113
BUILD_ID=18244.291.9

The runner comes online but while executing it gives the below error:

[2025-04-15 06:31:21,402] Looking for auth in config for image Image{name=docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable, runAsUser=None, auth=None} and found auth null

[2025-04-15 06:31:21,624] Pulling image k8s-docker.packages.atlassian.com/pause:3.8.

[2025-04-15 06:31:21,625] Looking for auth in config for image Image{name=k8s-docker.packages.atlassian.com/pause:3.8, runAsUser=None, auth=None} and found auth null

[2025-04-15 06:31:21,846] Removing container (name: e5e949ea-55c3-55d1-8fe5-15fbcbc06301_d08ed6b8-b01a-450b-8563-f87313b57bc9_pause)

[2025-04-15 06:31:21,860] Updating step progress to CLONING.

[2025-04-15 06:31:21,861] Creating container (name: e5e949ea-55c3-55d1-8fe5-15fbcbc06301_d08ed6b8-b01a-450b-8563-f87313b57bc9_pause).

[2025-04-15 06:31:21,981] {"traceId":"67fdfc5f1baf383dfd9bb17846b83eb4","parentId":"b73bae3e94343b4b","id":"93c4c32e89c5f5b3","kind":"CLIENT","name":"POST","timestamp":1744698681863042,"duration":117913,"localEndpoint":{"serviceName":"runner","ipv4":"172.16.1.219"},"tags":{"http.method":"POST","http.path":"/ex/bitbucket-pipelines/rest/1.0/accounts/{87642016-516a-46b5-b27d-1c58450bace4}/repositories/{c0c78c96-e5e7-42b5-be79-c337d17dd794}/pipelines/{3c6be4df-78d1-4696-a666-fb88e8472786}/steps/{d08ed6b8-b01a-450b-8563-f87313b57bc9}/progressUpdate"}}

[2025-04-15 06:31:21,984] Generating clone script.

[2025-04-15 06:31:22,018] Creating container (name: e5e949ea-55c3-55d1-8fe5-15fbcbc06301_d08ed6b8-b01a-450b-8563-f87313b57bc9_clone).

[2025-04-15 06:31:22,019] Executing clone script in clone container.

[2025-04-15 06:31:22,421] Container created (id: 3bb51d5ee96678e9ef328473ae301ee2fbce635161d84672f3a2e19a3eee8f36, name: e5e949ea-55c3-55d1-8fe5-15fbcbc06301_d08ed6b8-b01a-450b-8563-f87313b57bc9_pause).

[2025-04-15 06:31:22,423] Starting container (id: 3bb51d5ee96678e9ef328473ae301ee2fbce635161d84672f3a2e19a3eee8f36).

[2025-04-15 06:31:22,423] Container created (id: 54d07bc4670fb475b2221c3317a5a9a1cc8c2868907a8a26a659a901bafad6e2, name: e5e949ea-55c3-55d1-8fe5-15fbcbc06301_d08ed6b8-b01a-450b-8563-f87313b57bc9_clone).

[2025-04-15 06:31:22,427] Starting container (id: 54d07bc4670fb475b2221c3317a5a9a1cc8c2868907a8a26a659a901bafad6e2).

[2025-04-15 06:31:23,009] Removing container (name: e5e949ea-55c3-55d1-8fe5-15fbcbc06301_d08ed6b8-b01a-450b-8563-f87313b57bc9_system_auth-proxy)

[2025-04-15 06:31:23,018] Creating container (name: e5e949ea-55c3-55d1-8fe5-15fbcbc06301_d08ed6b8-b01a-450b-8563-f87313b57bc9_system_auth-proxy).

[2025-04-15 06:31:23,083] Adding container log: /var/lib/docker/containers/54d07bc4670fb475b2221c3317a5a9a1cc8c2868907a8a26a659a901bafad6e2/54d07bc4670fb475b2221c3317a5a9a1cc8c2868907a8a26a659a901bafad6e2-json.log

[2025-04-15 06:31:23,105] Waiting on container (id: 54d07bc4670fb475b2221c3317a5a9a1cc8c2868907a8a26a659a901bafad6e2) to exit.

[2025-04-15 06:31:23,113] Creating exec into container (id: 54d07bc4670fb475b2221c3317a5a9a1cc8c2868907a8a26a659a901bafad6e2).

[2025-04-15 06:31:23,166] Container created (id: 4a318f9f23d2fc9fa5c4fc4334f93018598614b54b50f93c5f003460381ca824, name: e5e949ea-55c3-55d1-8fe5-15fbcbc06301_d08ed6b8-b01a-450b-8563-f87313b57bc9_system_auth-proxy).

[2025-04-15 06:31:23,169] Create container warning: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable: Your kernel does not support memory swappiness capabilities or the cgroup is not mounted. Memory swappiness discarded.

[2025-04-15 06:31:23,170] Starting container (id: 4a318f9f23d2fc9fa5c4fc4334f93018598614b54b50f93c5f003460381ca824).

[2025-04-15 06:31:23,397] Adding container log: /var/lib/docker/containers/4a318f9f23d2fc9fa5c4fc4334f93018598614b54b50f93c5f003460381ca824/4a318f9f23d2fc9fa5c4fc4334f93018598614b54b50f93c5f003460381ca824-json.log

[2025-04-15 06:31:23,399] Waiting on container (id: 4a318f9f23d2fc9fa5c4fc4334f93018598614b54b50f93c5f003460381ca824) to exit.

[2025-04-15 06:31:23,489] Finished waiting on container (id: 54d07bc4670fb475b2221c3317a5a9a1cc8c2868907a8a26a659a901bafad6e2, exitResult: 1).

[2025-04-15 06:31:23,490] An error occurred whilst creating container exec in container (id: 54d07bc4670fb475b2221c3317a5a9a1cc8c2868907a8a26a659a901bafad6e2).

com.github.dockerjava.api.exception.ConflictException: Status 409: {"message":"Container 54d07bc4670fb475b2221c3317a5a9a1cc8c2868907a8a26a659a901bafad6e2 is not running"}

at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:101)

at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:32)

at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)

Kindly note, the docker command when ran from a local mac machine it is working fine and able to execute the pipeline fine.

But facing issue only with GKE cluster machine.

Let me know if any more info is required.