Created a pipe to use in next steps, but installed pip pakcages not found

[Elgiz Abbasov]

Hi everyone, so my pipeline looks like this:

image: atlassian/default-image:2

solved

[Igor Stoyanov]

@Elgiz Abbasov Hi. Happy that you solved your issue. 

To discover more pipes, visit our Bitbucket Pipes Marketplace.

Regards, Igor. 

[Elgiz Abbasov]

Hey Igor, just a quick question as a follow up, I am using a Dockerfile that has some packages installing using apt-get. In my BB pipelines even though I have set up Docker as my cache, it only caches the first 3 steps which are the least time consuming (env var setup) and the apt-get installs are not cached at all. Do you know how can I fix this so that I don't have to build the image from scratch in my pipeline step each time? Reading online people said Dockerfile should not contain apt-get installs (that there is a better way?) I know I can use DockerHub and update the base image everytime but I don't want to do that. Currently my pipeline takes ~10 minutes and it doesn't cache the Docker pip downloads using pipenv nor does it cache the apt-get installs. Any help would be greatly areciated

[Igor Stoyanov]

@Elgiz Abbasov Hi. Please provide us your Dockerfile so we could help you.

Regards, Igor.

[Elgiz Abbasov]

@Igor Stoyanov My Dockerfile:

FROM python:3.9

# Set environment variables

ENV PYTHONDONTWRITEBYTECODE 1

ENV PYTHONUNBUFFERED 1

ENV PYTHONPATH "${PYTHONPATH}:/opt/atlassian/pipelines/agent/build"

# Set work directory

WORKDIR /code

# Install dependencies

COPY Pipfile Pipfile.lock /code/

RUN pip install pipenv && pipenv install --dev --system

RUN apt-get update && apt-get autoclean && \

 apt-get install -y '^libxcb.*-dev' libx11-xcb-dev libglu1-mesa-dev libxrender-dev libxi-dev libxcb-xinerama0 libxkbcommon-dev libxkbcommon-x11-dev libxkbcommon-x11-0 libdbus-1-3 xvfb

# Copy project

COPY . /code/

[Elgiz Abbasov]

@Igor Stoyanov My partial bitbucket-pipelines.yml file:

It builds the image from scratch every time (only caching env vars in Dockerfile) and taking a lot of time to install apt-get and pip packages in Dockerfile. I thought adding caches: docker would cache the image but it doesn't, instead the image is built and saved in tmp-image.docker (from scratch everytime). I want it so that if Pipfile or Dockerfile hasn't changed, it just uses the tmp-image.docker from previous pipeline run, but again caches: docker didn't help me with that.

image: atlassian/default-image:2

options:

docker: true

pipelines:

pull-requests:

'**': #this runs as default for any branch not elsewhere defined

 - step: 

name: 'Build Docker and Setup'

caches:

 - docker

 - pip

script:

 - docker build -t elgiz/app .

 - docker save --output tmp-image.docker elgiz/app

artifacts:

 - tmp-image.docker

 - parallel:

 - step:

name: 'PyTest'

script:

 - docker load --input ./tmp-image.docker

 - echo "PyTest running..."

 - docker run --init -v $BITBUCKET_CLONE_DIR:/opt/atlassian/pipelines/agent/build/app elgiz/app xvfb-run pytest --cov=test/

 - echo "PyTest completed."

[Igor Stoyanov]

@Elgiz Abbasov hi.

Maybe its good idea:

1. to create your own docker image with all needed dependencies
2. or to find in docker hub image that already has most of needed dependencies that you need

Regards, Igor

[Elgiz Abbasov]

@Igor Stoyanov The goal was to avoid using DockerHub and use bitbucket pipelines features but I guess the caching doesn't work as I expected ;/ Thanks