Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

buildx push fails when repository blocks tag mutations #663

Closed
michaelb990 opened this issue Jul 1, 2021 · 4 comments
Closed

buildx push fails when repository blocks tag mutations #663

michaelb990 opened this issue Jul 1, 2021 · 4 comments

Comments

@michaelb990
Copy link

Hi folks!

I was testing a multi-architecture image push using buildx against an ECR repository with tag mutability set to IMMUTABLE. This blocks existing tags from being updated to point to new images. It seems as though buildx will push each image and the image index to the same tag, which causes the push to fail when immutability is on. I'm curious if it's possible for buildx to push the image manifests without a tag (using the manifest digest as the reference in the push URL). I believe most registries should support pushes by digest -- it's listed in the distribution spec this way. If not, would it be possible to add a flag to tell buildx optionally to skip tags on the "child" images, and only add a tag to the final image index?

I've added repro steps below. I also work at AWS on ECR, so happy to help answer questions on how tag mutability works if there ends up being any confusion.

Dockerfile:

❯ cat Dockerfile
FROM ubuntu

RUN touch helloworld.txt

Push to repository w/ mutability set to IMMUTABLE. This fails.

❯ docker buildx build --platform linux/amd64,linux/arm/v7 -t XXXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/immutable:mytag --push .
[+] Building 3.1s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                           0.0s
 => => transferring dockerfile: 31B                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                0.0s
 => [linux/arm/v7 internal] load metadata for docker.io/library/ubuntu:latest                                                                                  2.1s
 => [linux/amd64 internal] load metadata for docker.io/library/ubuntu:latest                                                                                   2.1s
 => [linux/amd64 1/2] FROM docker.io/library/ubuntu@sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe                                    0.0s
 => => resolve docker.io/library/ubuntu@sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe                                                0.0s
 => [linux/arm/v7 1/2] FROM docker.io/library/ubuntu@sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe                                   0.0s
 => => resolve docker.io/library/ubuntu@sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe                                                0.0s
 => CACHED [linux/arm/v7 2/2] RUN touch helloworld.txt                                                                                                         0.0s
 => CACHED [linux/amd64 2/2] RUN touch helloworld.txt                                                                                                          0.0s
 => ERROR exporting to image                                                                                                                                   0.8s
 => => exporting layers                                                                                                                                        0.0s
 => => exporting manifest sha256:40e84d26721b5ee0fde425c008e09cc279ba3b01b593f53cca2a3a386aff6c31                                                              0.0s
 => => exporting config sha256:ff73c6a722f2cabdb9eb484b5cdf98be10a3ac804ecd83b9a2f5390f797d7ae0                                                                0.0s
 => => exporting manifest sha256:2298d8347a50a4a9bf1ad7f3d30af2ef23802d22afe6e93251e14b4515dc1d7d                                                              0.0s
 => => exporting config sha256:fb83261d2895835d5238f59d1ad37384564781114c5369e9690e636e90c65a15                                                                0.0s
 => => exporting manifest list sha256:e3e2e98c9a8339fbec02265d89e88b52af79e6f139559a125b956abeb1a885f9                                                         0.0s
 => => pushing layers                                                                                                                                          0.4s
 => => pushing manifest for XXXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/immutable:mytag                                                                       0.4s
 => [auth] sharing credentials for XXXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com                                                                                0.0s
------
 > exporting to image:
------
error: failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:2298d8347a50a4a9bf1ad7f3d30af2ef23802d22afe6e93251e14b4515dc1d7d": unexpected status: 400 Bad Request

Set repository to MUTABLE.

❯ aws ecr put-image-tag-mutability --repository-name immutable --image-tag-mutability MUTABLE
{
    "registryId": "XXXXXXXXXXXX",
    "repositoryName": "immutable",
    "imageTagMutability": "MUTABLE"
}

Push to repository w/ mutability set to MUTABLE. This push succeeds.

❯ docker buildx build --platform linux/amd64,linux/arm/v7 -t XXXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/immutable:mytag --push .
[+] Building 3.5s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                           0.0s
 => => transferring dockerfile: 31B                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                0.0s
 => [linux/arm/v7 internal] load metadata for docker.io/library/ubuntu:latest                                                                                  2.4s
 => [linux/amd64 internal] load metadata for docker.io/library/ubuntu:latest                                                                                   2.2s
 => [linux/arm/v7 1/2] FROM docker.io/library/ubuntu@sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe                                   0.0s
 => => resolve docker.io/library/ubuntu@sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe                                                0.0s
 => [linux/amd64 1/2] FROM docker.io/library/ubuntu@sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe                                    0.0s
 => => resolve docker.io/library/ubuntu@sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe                                                0.0s
 => CACHED [linux/arm/v7 2/2] RUN touch helloworld.txt                                                                                                         0.0s
 => CACHED [linux/amd64 2/2] RUN touch helloworld.txt                                                                                                          0.0s
 => exporting to image                                                                                                                                         1.0s
 => => exporting layers                                                                                                                                        0.0s
 => => exporting manifest sha256:40e84d26721b5ee0fde425c008e09cc279ba3b01b593f53cca2a3a386aff6c31                                                              0.0s
 => => exporting config sha256:ff73c6a722f2cabdb9eb484b5cdf98be10a3ac804ecd83b9a2f5390f797d7ae0                                                                0.0s
 => => exporting manifest sha256:2298d8347a50a4a9bf1ad7f3d30af2ef23802d22afe6e93251e14b4515dc1d7d                                                              0.0s
 => => exporting config sha256:fb83261d2895835d5238f59d1ad37384564781114c5369e9690e636e90c65a15                                                                0.0s
 => => exporting manifest list sha256:e3e2e98c9a8339fbec02265d89e88b52af79e6f139559a125b956abeb1a885f9                                                         0.0s
 => => pushing layers                                                                                                                                          0.3s
 => => pushing manifest for XXXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/immutable:mytag                                                                       0.7s
 => [auth] sharing credentials for XXXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com                                                                                0.0s

Thanks!
@tonistiigi
Copy link
Member

This was fixed in moby/buildkit#2020

@michaelb990
Copy link
Author

Thanks, that's great news! If anyone else hits this, I did need to create a builder with the latest changes to pick this up. I ran:

docker buildx create --driver-opt image=moby/buildkit:master --use

And after that, it worked like a charm 🎉

@carocad
Copy link

carocad commented Mar 20, 2024

Not sure if this is the right place but this doesn't work for me when using containerd on Linux :(

@thaJeztah
Copy link
Member

@carocad might be better to open a new ticket with defaults about your setup and how to reproduce (but you can add a link to this ticket for reference)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tonistiigi @michaelb990 @thaJeztah @carocad