Workspace Runner Errors (OOM)

[Elias Balasis]

I followed the instructions under https://support.atlassian.com/bitbucket-cloud/docs/set-up-and-use-runners-for-linux/ to setup my Docker host and successfully installed a Pipeline Runner container.

However, when I run a pipeline I am getting the following in Docker console:

time="2022-04-15T04:37:39.207392198Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2022-04-15T04:37:39.207992940Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:2375"
time="2022-04-15T04:37:39.208030244Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:2375"
time="2022-04-15T04:37:40.208288526Z" level=warning msg="Binding to an IP address without --tlsverify is deprecated. Startup is intentionally being slowed down to show this message" host="tcp://0.0.0.0:2375"
time="2022-04-15T04:37:40.208370711Z" level=warning msg="Please consider generating tls certificates with client validation to prevent exposing unauthenticated root access to your network" host="tcp://0.0.0.0:2375"
time="2022-04-15T04:37:40.208404756Z" level=warning msg="You can override this by explicitly specifying '--tls=false' or '--tlsverify=false'" host="tcp://0.0.0.0:2375"
time="2022-04-15T04:37:40.208423146Z" level=warning msg="Support for listening on TCP without authentication or explicit intent to run without authentication will be removed in the next release" host="tcp://0.0.0.0:2375"
time="2022-04-15T04:37:55.298287690Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2022-04-15T04:37:55.300465539Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2022-04-15T04:37:55.329734002Z" level=warning msg="failed to load plugin io.containerd.internal.v1.opt" error="mkdir /opt/containerd: read-only file system"
time="2022-04-15T04:37:55.445632118Z" level=warning msg="Your kernel does not support cgroup blkio weight"
time="2022-04-15T04:37:55.445697874Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
time="2022-04-15T04:37:55.465594681Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'\nbridge 200704 1 br_netfilter\nstp 16384 1 bridge\nllc 16384 2 bridge,stp\nip: can't find device 'br_netfilter'\nbr_netfilter 24576 0 \nbridge 200704 1 br_netfilter\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1"
time="2022-04-15T04:37:55.468936072Z" level=warning msg="Running iptables --wait -t nat -L -n failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory\niptables v1.8.6 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.`, error: exit status 3"
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: modprobe: can't change directory to '/lib/modules': No such file or directory
iptables v1.8.6 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)

I am also getting the following in my runner's Docker container logs:

[2022-04-15 04:35:17,618] Runner version: 1.315
[2022-04-15 04:35:17,687] Runner runtime: linux-docker
[2022-04-15 04:35:22,878] Copying Docker cli to working directory.
[2022-04-15 04:35:24,491] Starting websocket listening to RUNNER_UPDATED events.
[2022-04-15 04:35:24,761] Updating runner status to "ONLINE" and checking for new steps assigned to the runner after 0 seconds and then every 30 seconds.
[2022-04-15 04:35:25,559] Updating runner state to "ONLINE".
[2022-04-15 04:35:54,771] Updating runner state to "ONLINE".
[2022-04-15 04:35:59,123] Setting runner state to executing step.
[2022-04-15 04:35:59,163] Getting step StepId{accountUuid={3e496ad3-eaeb-4e1a-8787-64eba94fb93a}, repositoryUuid={141de12d-e095-41d0-9a88-1a8dac481c11}, pipelineUuid={6b748fbb-a984-4d4a-9d91-2db9150a9b5b}, stepUuid={87ec6596-c15e-433c-ac9b-0bd08f506bcd}}.
[2022-04-15 04:35:59,179] Getting oauth token for step.
[2022-04-15 04:35:59,191] Getting environment variables for step.
[2022-04-15 04:36:00,062] Getting all artifacts for step.
[2022-04-15 04:36:00,076] Getting SSH private key.
[2022-04-15 04:36:00,081] Getting known hosts.
[2022-04-15 04:36:00,419] SSH private key not found
[2022-04-15 04:36:00,744] Setting up directories.
[2022-04-15 04:36:00,780] Starting log uploader.
[2022-04-15 04:36:00,822] Removing container 455c0d9a-fbce-5c76-ad8f-764ede27b451_87ec6596-c15e-433c-ac9b-0bd08f506bcd_clone
[2022-04-15 04:36:00,945] Removing container 455c0d9a-fbce-5c76-ad8f-764ede27b451_87ec6596-c15e-433c-ac9b-0bd08f506bcd_build
[2022-04-15 04:36:01,001] Setting up step timeout: PT2H
[2022-04-15 04:36:01,004] Starting websocket listening to STEP_COMPLETED events.
[2022-04-15 04:36:01,007] Checking for step completion every PT30S seconds.
[2022-04-15 04:36:01,260] Updating step progress to PULLING_IMAGES.
[2022-04-15 04:36:01,545] Pulling image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-dvcs-tools:prod-stable.
[2022-04-15 04:36:01,837] Appending log line to main log.
[2022-04-15 04:36:12,307] Pulling image maven:3.6.3.
[2022-04-15 04:36:24,843] Updating runner state to "ONLINE".
[2022-04-15 04:36:54,773] Updating runner state to "ONLINE".
[2022-04-15 04:37:06,825] Pulling image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable.
[2022-04-15 04:37:14,378] Pulling image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v20.10.6-prod-stable.
[2022-04-15 04:37:24,770] Updating runner state to "ONLINE".
[2022-04-15 04:37:33,085] Pulling image docker-hub.packages.atlassian.com/google/pause:latest.
[2022-04-15 04:37:36,719] Removing container 455c0d9a-fbce-5c76-ad8f-764ede27b451_87ec6596-c15e-433c-ac9b-0bd08f506bcd_pause
[2022-04-15 04:37:36,728] Updating step progress to CLONING.
[2022-04-15 04:37:36,729] Creating container 455c0d9a-fbce-5c76-ad8f-764ede27b451_87ec6596-c15e-433c-ac9b-0bd08f506bcd_pause.
[2022-04-15 04:37:36,960] Generating clone script.
[2022-04-15 04:37:37,015] Creating container 455c0d9a-fbce-5c76-ad8f-764ede27b451_87ec6596-c15e-433c-ac9b-0bd08f506bcd_clone.
[2022-04-15 04:37:37,015] Executing clone script in clone container.
[2022-04-15 04:37:37,405] Starting container.
[2022-04-15 04:37:37,414] Starting container.
[2022-04-15 04:37:38,493] Removing container 455c0d9a-fbce-5c76-ad8f-764ede27b451_87ec6596-c15e-433c-ac9b-0bd08f506bcd_system_auth-proxy
[2022-04-15 04:37:38,501] Creating container 455c0d9a-fbce-5c76-ad8f-764ede27b451_87ec6596-c15e-433c-ac9b-0bd08f506bcd_system_auth-proxy.
[2022-04-15 04:37:38,501] Removing container 455c0d9a-fbce-5c76-ad8f-764ede27b451_87ec6596-c15e-433c-ac9b-0bd08f506bcd_system_docker
[2022-04-15 04:37:38,525] Creating container 455c0d9a-fbce-5c76-ad8f-764ede27b451_87ec6596-c15e-433c-ac9b-0bd08f506bcd_system_docker.
[2022-04-15 04:37:38,543] Adding container log: /var/lib/docker/containers/e611aabb4640a049664a1f03e891ef9ec1ec6417b5eea1ba7cf1fa413223cc9f/e611aabb4640a049664a1f03e891ef9ec1ec6417b5eea1ba7cf1fa413223cc9f-json.log
[2022-04-15 04:37:38,566] Waiting on container to exit.
[2022-04-15 04:37:38,583] Creating exec into container.
[2022-04-15 04:37:38,610] Starting exec into container and waiting for exec to exit.
[2022-04-15 04:37:38,707] Starting container.
[2022-04-15 04:37:38,729] Starting container.
[2022-04-15 04:37:39,182] Adding container log: /var/lib/docker/containers/4ea7606f9fc4bc322c9b959a041357abc1567530adaeeffcb83149e83249d1fe/4ea7606f9fc4bc322c9b959a041357abc1567530adaeeffcb83149e83249d1fe-json.log
[2022-04-15 04:37:39,185] Waiting on container to exit.
[2022-04-15 04:37:39,209] Adding container log: /var/lib/docker/containers/eeda36321bda31bcc55c092b28ae953dc146819448fc45047c025f453baa4e7b/eeda36321bda31bcc55c092b28ae953dc146819448fc45047c025f453baa4e7b-json.log
[2022-04-15 04:37:39,269] Waiting on container to exit.
[2022-04-15 04:37:39,827] Appending log line to main log.
[2022-04-15 04:37:39,834] Appending log line to log: {193c9579-0796-45f0-8cf4-fc5348abaaa4}.
[2022-04-15 04:37:40,155] Appending log line to log: {43756c51-c100-43a9-bc30-06f4b46e11cd}.
[2022-04-15 04:37:40,822] Appending log line to log: {43756c51-c100-43a9-bc30-06f4b46e11cd}.
[2022-04-15 04:37:41,484] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:41,487] Not downloading artifacts. (artifactsDownloadEnabled: true, numberOfArtifacts: 0)
[2022-04-15 04:37:41,487] Updating step progress to DOWNLOADING_CACHES.
[2022-04-15 04:37:41,719] Getting content uri for cache.
[2022-04-15 04:37:41,833] Appending log line to main log.
[2022-04-15 04:37:41,931] Cache not found.
[2022-04-15 04:37:41,938] Inspecting image maven:3.6.3.
[2022-04-15 04:37:41,960] Inspecting image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v20.10.6-prod-stable.
[2022-04-15 04:37:41,997] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:42,004] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:42,815] Appending log line to main log.
[2022-04-15 04:37:42,986] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:43,986] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:44,986] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:45,986] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:46,991] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:47,991] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:48,984] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:49,987] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:50,988] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:51,983] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:52,986] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:54,003] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:54,766] Updating runner state to "ONLINE".
[2022-04-15 04:37:54,992] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:55,825] Appending log line to log: {43756c51-c100-43a9-bc30-06f4b46e11cd}.
[2022-04-15 04:37:55,986] Container has state (exitCode: Some(0), OOMKilled Some(false))
[2022-04-15 04:37:56,685] Container has state (exitCode: Some(1), OOMKilled Some(false))
[2022-04-15 04:37:56,706] Removing container 455c0d9a-fbce-5c76-ad8f-764ede27b451_87ec6596-c15e-433c-ac9b-0bd08f506bcd_build
[2022-04-15 04:37:56,713] Not uploading caches. (numberOfCaches: 1, resultOrError: FAILED)
[2022-04-15 04:37:56,717] Updating step progress to UPLOADING_ARTIFACTS.
[2022-04-15 04:37:56,813] Appending log line to main log.
[2022-04-15 04:37:56,820] Appending log line to log: {43756c51-c100-43a9-bc30-06f4b46e11cd}.

[Elias Balasis · Accepted solution]

This finally doesn't seem to be an actual problem.

The original issue was a side effect of something else.

Please archive this case.