Hi,
we use bitbucket pipelines for our CD to Google cloud.
Among others, we build an apache image. Inside it's dockefile we have a step to remove default apache's vhosts:
RUN rm -rf /etc/apache2/sites-available/*
This worked fine until yesterday. Today the rm stopped working and after searching we found out that it's probably due to bitbucket's filesystem which is formatted without d_type support.
*We deployed the image outside the pipeline properly, so we're sure that it's related to bitbucket.
Can you advice a workaround for this?
Thanks
Hello,
We've rolled out some changes that have fixed this issue. Can you please try run your docker builds again. Please comment if you're still having issues with deleting/editing files as part of your Docker build.
Thanks,
Phil
EDIT: This is scoped to any deletion. Not just symlinks.
Hello,
The Pipelines team has done some investigation into this, and this issue seems to be happening when the following three conditions are met:
We're still looking for a long term fix for this, but in the meantime you can try the following workarounds:
An example of the Docker build workaround. Is starting with this (failing) Docker image:
FROM ubuntu:18.04
RUN echo "test" > file1.txt
# Shows file1.txt
RUN ls -l
# Attempt to delete file1.txt
RUN rm file1.txt
# Shows that file1.txt still exists. When it should have been deleted.
RUN ls -l
You can then change it to instead be:
FROM ubuntu:18.04
RUN echo "test" > file1.txt \
&& ls -l \
&& rm file1.txt \
&& ls -l
Sorry for the inconvenience. We're still pinpointing the root cause of this.
@Tania Petsouka, your account has been whitelisted (as mentioned in your support ticket). You should be able to run pipelines again without any other workarounds.
Thanks,
Phil
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Philip Hodder We are experiencing the same issue since the beginning of October.
Our build is based on the official docker image tomcat:8.5-jre8. Our Dockerfile removes all tomcat default apps with this command:
RUN rm -rf /usr/local/tomcat/webapps/
The build runs fine but the final docker image contains all default webapps which should have been deleted. This worked until the beginning of October.
I also think it could be related to d_type OverlayFS-Driver as supposed by @Tania Petsouka:
https://docs.docker.com/storage/storagedriver/overlayfs-driver/
Docker will not support running without d_type support in future releases.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi @Malte Hübner,
Can you open a support ticket for the team to investigate in more detail. Please ask the Support Engineer to ping me on Stride. So I can whitelist your account and see if that helps, before I go to sleep (it's currently 11pm here). I'll likely be offline in an hour.
I'm struggling to reproduce this problem in Pipelines. I used a fairly trivial Dockerfile to try and reproduce this.
FROM tomcat:8.5-jre8
RUN ls -l # Shows webapps directory
RUN rm -rf /usr/local/tomcat/webapps && ls -l # Shows webapps directory has been deleted
RUN ls -l # webapps directory is still deleted
I tried this with my repo both enabled, and disabled user namespace remapping.
Is there a Dockerfile you can provide that will reproduce the problem?
In regards to overlay and d_type support, if you run 'docker info' inside of Pipelines you'll see we're using overlay2 with d_type support.
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
None of the referenced issues seem to address overlay2, only overlay. So I'm unsure if that's the issue. Particularly due to the timing of these issues being raised. The only Docker-related change we made at the start of October (that I'm aware of) is the user namespace remapping being enabled.
If needed, I'll chase this up with the rest of the team tomorrow.
Thanks,
Phil
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thank you for your reply @Philip Hodder! I just tested the pipeline build and the problem seems to be gone. Your fix from Tuesday fixes my issue. Thanks again!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Ah! Brilliant! No problem. :) Glad to hear it's resolved.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Tania,
Do you have any news on that topic?
We experience the same issue. I already created a ticket.
As a workaround, you could use a customized base docker image, where you already deleted the vhosts.
br
Tobi
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Tobias,
i sent a ticket as well and initially they said that:
"Our team had enabled a security feature on the docker daemons to isolate containers with a user namespace.This update may have affected the build that you're experiencing now.For more info about the update: https://docs.docker.com/engine/security/userns-remap/"
Afterwards they said that they're still investigating.
I doubt that this is the issue. I am confident it's due to storage type "overlay" and/or d_type support:
https://stackoverflow.com/questions/42183994/dockerfile-rm-rf-fail
https://github.com/moby/moby/issues/31321
Still waiting by them.
Thanks for the workaround suggestion!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Tania Petsouka what exactly solved your problems? Creating the ticket for Atlassian support to add your account to whitelist or creating a new base image?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Malte,
yes that's what we did.
But according to Philip's post above (on Tuesday) i guessed that they found a more global solution. Not sure though.
I'd suggest that you open a ticket as well and add a link to this thread.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Indeed the problem is solved with the fix from Tuesday. I did not run another build after Tuesday. My fault! Thanks for your reply!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Online forums and learning are now in one easy-to-use experience.
By continuing, you accept the updated Community Terms of Use and acknowledge the Privacy Policy. Your public name, photo, and achievements may be publicly visible and available in search engines.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.