Forums

Articles
Create
cancel
Showing results for 
Search instead for 
Did you mean: 

rm -rf inside container via Bitbucket pipeline

Tania Petsouka September 28, 2018

Hi,

we use bitbucket pipelines for our CD to Google cloud.

Among others, we build an apache image. Inside it's dockefile we have a step to remove default apache's vhosts:

RUN rm -rf /etc/apache2/sites-available/*

This worked fine until yesterday. Today the rm stopped working and after searching we found out that it's probably due to bitbucket's filesystem which is formatted without d_type support.

See here http://www.rubyrube.com/2017/06/28/dockerfile-run-rm-not-working-ex-run-rm-rf-usrlocaltomcatwebappsroot/

*We deployed the image outside the pipeline properly, so we're sure that it's related to bitbucket.

Can you advice a workaround for this?

 

Thanks

 

 

 

4 answers

1 accepted

0 votes
Answer accepted
Philip Hodder
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 9, 2018

Hello,

We've rolled out some changes that have fixed this issue. Can you please try run your docker builds again. Please comment if you're still having issues with deleting/editing files as part of your Docker build.

Thanks,

Phil

2 votes
Philip Hodder
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 2, 2018

EDIT: This is scoped to any deletion. Not just symlinks.

Hello,

The Pipelines team has done some investigation into this, and this issue seems to be happening when the following three conditions are met:

  • Your account has user namespace remapping enabled. This is currently enabled globally. (This was a security feature we've wanted to enable for a while and significantly improves the security of our build cluster.)
  • You are trying to delete a file.
  • You are doing the deletion inside of Docker. (With a few additional conditions)

We're still looking for a long term fix for this, but in the meantime you can try the following workarounds:

  • Contact Atlassian Support. We'll add you to the whitelist to disable user namespace remapping (please reference this Community question so we can verify you fall into these known conditions).
  • Delete the directory or file in the same Docker build instruction as it's created.

An example of the Docker build workaround. Is starting with this (failing) Docker image:

FROM ubuntu:18.04
RUN echo "test" > file1.txt

# Shows file1.txt
RUN ls -l

# Attempt to delete file1.txt
RUN rm file1.txt

# Shows that file1.txt still exists. When it should have been deleted.
RUN ls -l

You can then change it to instead be:

FROM ubuntu:18.04
RUN echo "test" > file1.txt \
&& ls -l \
&& rm file1.txt \
&& ls -l

Sorry for the inconvenience. We're still pinpointing the root cause of this.

@Tania Petsouka, your account has been whitelisted (as mentioned in your support ticket). You should be able to run pipelines again without any other workarounds.

Thanks,

Phil

Tania Petsouka October 3, 2018

Cool, thanks!

0 votes
Malte Hübner
Contributor
October 11, 2018

@Philip Hodder We are experiencing the same issue since the beginning of October.

Our build is based on the official docker image tomcat:8.5-jre8. Our Dockerfile removes all tomcat default apps with this command:

RUN rm -rf /usr/local/tomcat/webapps/

The build runs fine but the final docker image contains all default webapps which should have been deleted. This worked until the beginning of October. 

I also think it could be related to d_type OverlayFS-Driver as supposed by @Tania Petsouka:  

https://docs.docker.com/storage/storagedriver/overlayfs-driver/

Docker will not support running without d_type support in future releases.

Philip Hodder
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 11, 2018

Hi @Malte Hübner,

Can you open a support ticket for the team to investigate in more detail. Please ask the Support Engineer to ping me on Stride. So I can whitelist your account and see if that helps, before I go to sleep (it's currently 11pm here). I'll likely be offline in an hour.

I'm struggling to reproduce this problem in Pipelines. I used a fairly trivial Dockerfile to try and reproduce this.

FROM tomcat:8.5-jre8
RUN ls -l # Shows webapps directory
RUN rm -rf /usr/local/tomcat/webapps && ls -l # Shows webapps directory has been deleted
RUN ls -l # webapps directory is still deleted

I tried this with my repo both enabled, and disabled user namespace remapping.

Is there a Dockerfile you can provide that will reproduce the problem?

In regards to overlay and d_type support, if you run 'docker info' inside of Pipelines you'll see we're using overlay2 with d_type support.

Storage Driver: overlay2 
Backing Filesystem: extfs

Supports d_type: true

Native Overlay Diff: true

None of the referenced issues seem to address overlay2, only overlay. So I'm unsure if that's the issue. Particularly due to the timing of these issues being raised. The only Docker-related change we made at the start of October (that I'm aware of) is the user namespace remapping being enabled.

If needed, I'll chase this up with the rest of the team tomorrow.

Thanks,

Phil

Malte Hübner
Contributor
October 11, 2018

Thank you for your reply @Philip Hodder! I just tested the pipeline build and the problem seems to be gone. Your fix from Tuesday fixes my issue. Thanks again!

Philip Hodder
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 11, 2018

Ah! Brilliant! No problem. :) Glad to hear it's resolved.

0 votes
torauter
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
October 1, 2018

Hi Tania,

Do you have any news on that topic?

We experience the same issue. I already created a ticket. 

As a workaround, you could use a customized base docker image, where you already deleted the vhosts.

 

br

Tobi

Tania Petsouka October 1, 2018

Hi Tobias,

i sent a ticket as well and initially they said that:

"Our team had enabled a security feature on the docker daemons to isolate containers with a user namespace.This update may have affected the build that you're experiencing now.For more info about the update: https://docs.docker.com/engine/security/userns-remap/"

 Afterwards they said that they're still investigating.

I doubt that this is the issue. I am confident it's due to storage type "overlay" and/or d_type support:

https://stackoverflow.com/questions/42183994/dockerfile-rm-rf-fail

https://github.com/moby/moby/issues/31321

 

Still waiting by them. 

Thanks for the workaround suggestion!

Malte Hübner
Contributor
October 11, 2018

@Tania Petsouka what exactly solved your problems? Creating the ticket for Atlassian support to add your account to whitelist or creating a new base image?

Tania Petsouka October 11, 2018

Hi Malte,

yes that's what we did. 

But according to Philip's post above (on Tuesday) i guessed that they found a more global solution. Not sure though.

I'd suggest that you open a ticket as well and add a link to this thread.

Malte Hübner
Contributor
October 11, 2018

Indeed the problem is solved with the fix from Tuesday. I did not run another build after Tuesday. My fault! Thanks for your reply!

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events