Pipelines - Failing to pull and unpack docker image (rpc error)

[Leonardo Gardel Valverde]

We have been using the same version of a docker hub repository as the image of our bitbucket pipelines for 1 year. Suddenly, it begun failing and we don't know the reason.

The code below is at the top of the bitbucket-pipelines.yml file

image:
  name: juryvox/juryvox-react-test
  username: $DOCKER_HUB_USERNAME
  password: $DOCKER_HUB_PASSWORD
  email: $DOCKER_HUB_EMAIL

The pipelines fails right after its beginning, with the following error.

rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/juryvox/juryvox-react-test:latest": failed to copy: httpReaderSeeker: failed open: unexpected status code http://localhost:5000/v2/juryvox/juryvox-react-test/blobs/sha256:80e2dc61b67de22acb940d4bea02b2cd973abc3e7fd2bfd2fe1a486ce2dfde6f: 500 Internal Server Error

I have already checked if I am able to pull the repository from dockerhub using the same username and password that are set as bitbucket's repository variables, and it worked. I don't understand why it became a problem without any change neither at the container repository, nor at the bitbucket-pipelines.yml.

What could be causing this error?

[Patryk Kwiatkowski]

similar issue since today:

rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/maven:3.3.3-jdk-7": failed to unpack image on snapshotter overlayfs: failed to extract layer sha256:42755cf4ee95900a105b4e33452e787026ecdefffcc1992f961aa286dc3f7f95: failed to get reader from content store: content digest sha256:5c90d4a2d1a8dfffd05ff2dd659923f0ca2d843b5e45d030e17abbcd06a11b5b: not found

[Patryk Kwiatkowski]

In our case we had to change docker image for maven from 3.3.3-jdk-7 to 3.3.9

[Antonio Jerez]

Hi there.

We are consistently experiencing this issue too, last time was a 7 hours window we couldn't deploy using any dockerhub image, but it randomly pops here and there...

[Daniel Silva Reis]

Hello Antonio!

We are experiencing the same issue. Have you found any solution?

[Robert Goldsmith]

This happens periodically for us and we have to wait about 10mins and then manually restart the pipeline. It's really annoying.

[Johnathan Mayke Melo Neto]

We're also experiencing this issue.

[Gerwin van de Steeg]

A similar problem just started intermittently happening for us using an AWS ECR as the backend.  Doesn't appear to be resolved.

[Michel Moreau]

It's been a 6 days window here... python:3.7.2 image

[Vinicius Valente]

I managed to solve it by changing the python version from 3.5.1 to 3.7.2

[Rohan Gore]

I am also facing the same issue since yesterday, though there are a couple of behaviours different in my case

1. Could not recall this happening before, have been running the same private image from Docker Hub for more than a year at least across different BB repos.

2. Restarting the pipeline does not make it work, and it is failing in all of these repos.

Note: Docker Hub Credentials are not changed. Able to login to Docker Hub using the same creds. Also used another pair of creds, still the same error!!

[Rohan Gore]

In my case,  it was happening as we were using the legacy Docker Hub plan. The moment we switched to the new team license, everything started working.

[Matt Cavanagh]

We're also experiencing this issue periodically. It appears you also can't reliably re-run the pipeline for a period when this issue occurs.

Errors are similar to above, except this is one of Atlassian's docker images.

[nicolas_nitche]

I am getting a similar error:

rpc error: code = Unknown desc = failed to pull and unpack image "company.azurecr.io/custom-image:latest": failed to copy: read tcp XX.XX.XX.XX:XXXX->XX.XX.XX.XX:XXX: read: connection timed out

No solution has been found yet. Like Anton, I need to restart the pipeline when the error occur, and it works. 
Does anyone found a clean solution on that ?

[Anton Daneyko]

I am getting a similar error:

```

rpc error: code = Unknown desc = failed to pull and unpack image "eu.gcr.io/lofty-seer-161814/othoz-docker-base:4.0.0": failed to resolve reference "eu.gcr.io/lofty-seer-161814/othoz-docker-base:4.0.0": failed to authorize: failed to fetch oauth token: Get https://eu.gcr.io/v2/token?scope=repository%3Alofty-seer-161814%2Fothoz-docker-base%3Apull&service=eu.gcr.io: dial tcp 173.194.66.82:443: i/o timeout

```

it happens sporadically.  A restart goes OK once in a while.

[Le Phuong]

@Graham Gatus We also see the same issue intermittently. A re-run would fix it at the moment but we hope this would be resolved completely.

[Graham Gatus]

@Leonardo Gardel Valverde are you still experiencing this issue? If so can you please raise a support case via https://support.atlassian.com so we can investigate further.

[Leonardo Gardel Valverde]

I had to make my docker image public to make it work. I think there is some bug with the communication between the repository variables and the pipelines instance. The code I posted worked for a long time, until I update my docker hub password. Then, I updated the repository variables with the new value and it stopped working, and the problem was not the password, because I copied pasted it on docker hub and it worked.

I hope my problem may help your team to find the bug, but I'm no longer interested in the answer.

Sorry for posting my problem at the wrong place.