Docker daemon start failure for self-hosted runner

[Elias Balasis]

On self-hosted runner, pipeline steps fail to start reporting the following:

time="2022-04-16T21:51:36.074513821Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2022-04-16T21:51:36.074957119Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:2375"
time="2022-04-16T21:51:36.074980513Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:2375"
time="2022-04-16T21:51:37.075347836Z" level=warning msg="Binding to an IP address without --tlsverify is deprecated. Startup is intentionally being slowed down to show this message" host="tcp://0.0.0.0:2375"
time="2022-04-16T21:51:37.075485146Z" level=warning msg="Please consider generating tls certificates with client validation to prevent exposing unauthenticated root access to your network" host="tcp://0.0.0.0:2375"
time="2022-04-16T21:51:37.075501936Z" level=warning msg="You can override this by explicitly specifying '--tls=false' or '--tlsverify=false'" host="tcp://0.0.0.0:2375"
time="2022-04-16T21:51:37.075526046Z" level=warning msg="Support for listening on TCP without authentication or explicit intent to run without authentication will be removed in the next release" host="tcp://0.0.0.0:2375"
time="2022-04-16T21:51:52.184158579Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2022-04-16T21:51:52.185239911Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2022-04-16T21:51:52.193368196Z" level=warning msg="failed to load plugin io.containerd.internal.v1.opt" error="mkdir /opt/containerd: read-only file system"
time="2022-04-16T21:51:52.290152805Z" level=warning msg="Your kernel does not support cgroup blkio weight"
time="2022-04-16T21:51:52.290199485Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
time="2022-04-16T21:51:52.309341368Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'\nbridge 200704 1 br_netfilter\nstp 16384 1 bridge\nllc 16384 2 bridge,stp\nip: can't find device 'br_netfilter'\nbr_netfilter 24576 0 \nbridge 200704 1 br_netfilter\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1"
time="2022-04-16T21:51:52.312787457Z" level=warning msg="Running iptables --wait -t nat -L -n failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory\niptables v1.8.6 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.`, error: exit status 3"
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: modprobe: can't change directory to '/lib/modules': No such file or directory
iptables v1.8.6 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)

The OS of the Docker host of the self-hosted runner is "AlmaLinux release 8.5 (Arctic Sphynx)"

I fear this relates to the docker-in-docker run for self-hosted runners.

[Elias Balasis · Accepted solution]

The legacy IP tables are not supported in bitbucket runners.

"modprobe ip_tables" adds a recent IP tables module to the OS kernel.

[Caroline R]

Hi, Elias, 

I saw that you created a ticket with our support related to the same question, so to avoid any possible misunderstandings or miscommunication that may arise from discussing the same matter on two different platforms, let’s focus on the ticket instead.

Once the ticket is resolved, feel free to share the resolution here to help other community members that may have the same issue.

Kind regards,
Caroline

[Elias Balasis]

see https://getsupport.atlassian.com/servicedesk/customer/portal/11/BBS-184376

[Elias Balasis]

The problem doesn't seem to be reproducible on a CentOS-7 Docker host.

This is definitely a bug, can someone raise this with Atlassian, as I am still on the free tier and I don't have permission to raise bugs.

see https://community.atlassian.com/t5/Bitbucket-questions/Self-hosted-runner-step-set-up-fails-when-trying-to-use-docker/qaq-p/1764689 (Nov. 17, 2021)