Selenium Tests Passing in Docker Locally But Failing in Bitbucket Pipeline

[Jonathan Buys]

I have a Django project that we are running Selenium UI tests for. I have the tests running locally in Docker, but the same tests are failing when run through BitBucket Pipelines. 

The relevant portion of the Pipeline config looks like this:

image: python:2
pipelines:
default:
- step:
caches:
- pip
script: # Modify the commands below to build your repository.
- echo "Simple changes kick off pipelines!"
branches:
chrome-headless-bitbucket:
- step:
caches:
- pip

script:
- apt-get update
- apt-get install -y libxss1 libappindicator1 libindicator7 libxi6 libgconf-2-4
- wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
- dpkg -i google-chrome*.deb || true
- apt-get install -f -y
- apt-get install -y xvfb unzip
- wget -N https://chromedriver.storage.googleapis.com/74.0.3729.6/chromedriver_linux64.zip
- unzip chromedriver_linux64.zip && chmod +x chromedriver
- mv -f chromedriver /usr/local/share/chromedriver
- ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
- ln -s /usr/local/share/chromedriver /usr/bin/chromedriver
- pip install pyvirtualdisplay selenium
- cd tsmartweb && pip install -r requirements.txt && pip install -r requirements-dev.txt
- pytest

And the corresponding Dockerfile looks like this:

FROM python:2

RUN apt-get update
RUN apt-get install -y libxss1 libappindicator1 libindicator7 libxi6 libgconf-2-4 xvfb unzip
RUN wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

RUN dpkg -i google-chrome*.deb || true
RUN apt-get install -f -y

RUN wget -N https://chromedriver.storage.googleapis.com/74.0.3729.6/chromedriver_linux64.zip
RUN unzip chromedriver_linux64.zip
RUN chmod +x chromedriver

RUN mv -f chromedriver /usr/local/share/chromedriver
RUN ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
RUN ln -s /usr/local/share/chromedriver /usr/bin/chromedriver
RUN pip install pyvirtualdisplay selenium

RUN mkdir -p /usr/local/src/
ADD . /usr/local/src/
RUN cd /usr/local/src/projectName/ && pip install -r requirements.txt
RUN cd /usr/local/src/projectName/ && pip install -r requirements-dev.txt

CMD ["pytest"]

To test I've been building the image, and running it with this command:

docker run -t -i --volume=/Users/username/.aws:/root/.aws --memory=4g --memory-swap=4g --memory-swappiness=0 --workdir="/usr/local/src/projectName" 4aec08b23718 pytest

Most of my tests pass locally, but none pass in Bitbucket. I'm getting this error in the console:

self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7f595777c410>response = {'sessionId': 'd30ab389776d7ad689bbaa055f460cbf', 'status': 7, 'value': {'message': 'no such element: Unable to locate...9.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729@{#29}),platform=Linux 4.19.23-coreos-r1 x86_64)'}}

...

>       raise exception_class(message, screen, stacktrace)E       NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"navbar-username"}E         (Session info: headless chrome=74.0.3729.131)E         (Driver info: chromedriver=74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729@{#29}),platform=Linux 4.19.23-coreos-r1 x86_64)

Why would tests pass in my local Docker environment, but not in Pipelines?

- macOS version 10.14.4 (18E226)

- Docker version 18.09.2, build 6247962

[Jonathan Buys]

Of course, as soon as I post this the answer comes becomes apparent. IAM permission differences between my keys I'm using to test locally and the keys used in Bitbucket environmental variables. 

If anyone else runs into this issue, I was able to diagnose the problem by downloading the screenshots that Selenium takes on failure as artifacts, as defined in the Pipelines config. 

[Alexander]

Many thanks! Your YAML is something that helped me a lot to start using pipelines and selenium tests on top of it!