You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Andrei Sekretenko (Jira)" <ji...@apache.org> on 2020/02/03 18:07:00 UTC
[jira] [Commented] (MESOS-10092) Cannot pull image from docker
registry which does not reply with 'scope'/'service' in WWW-Authenticate
header
[ https://issues.apache.org/jira/browse/MESOS-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029146#comment-17029146 ]
Andrei Sekretenko commented on MESOS-10092:
-------------------------------------------
Briefly discussed with [~vinodkone] and [~abudnik]; we came to conclusion that the safest approach to fix this is the following:
First, we query the repository the old way, with the manifest/blob URI.
If it replies with both three of service, scope and realm, we assume that they know what they are doing and proceed with building auth server URI as before.
If the initial `WWW-Authenticate` is missing any of the three, then, instead of failing, we re-send the initial query to the repository root URI (nvcr.io/v2 in this example), get the realm from the response and compose the scope ourselves (like Docker does).
> Cannot pull image from docker registry which does not reply with 'scope'/'service' in WWW-Authenticate header
> -------------------------------------------------------------------------------------------------------------
>
> Key: MESOS-10092
> URL: https://issues.apache.org/jira/browse/MESOS-10092
> Project: Mesos
> Issue Type: Bug
> Reporter: Andrei Sekretenko
> Assignee: Andrei Sekretenko
> Priority: Critical
>
> This problem was encountered when trying to specify container image nvcr.io/nvidia/tensorflow:19.12-tf1-py3
> When initiating Docker Registry authentication (https://docs.docker.com/registry/spec/auth/token/) with nvcr.io, Mesos URI fetcher receives 'WWW-Authenticate' header without 'service' and 'scope' params, and fails here:
> https://github.com/apache/mesos/blob/1e9b121273a6d9248a78ab44798bd4c1138c31ee/src/uri/fetchers/docker.cpp#L1083
> This is an example of an unsuccessful request made by Mesos:
> {code}
> curl -s -S -L -i --raw --http1.1 -H "Accept: application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws" -y 60 https://nvcr.io/v2/nvidia/tensorflow/manifests/19.08-py3
> HTTP/1.1 401 Unauthorized
> Content-Type: text/html
> Date: Wed, 22 Jan 2020 19:01:57 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth?scope=repository:nvidia/tensorflow:pull,push"
> Content-Length: 195
> Connection: keep-alive
> <html>
> <head><title>401 Authorization Required</title></head>
> <body bgcolor="white">
> <center><h1>401 Authorization Required</h1></center>
> <hr><center>nginx/1.14.2</center>
> </body>
> </html>
> {code}
> At the same time, docker is perfectly capable of pulling this image.
> Note that the document "Token Authentication Specification" (https://docs.docker.com/registry/spec/auth/token/), on which the Mesos implementation is based, is vague on the issue of registries that do not provide 'scope'/'service' in WWW-Authenticate header.
> What Docker does differently (at the very least, in the case of nvcr.io):
> It sends the initial request not to the maniferst/blob URI, but to the repository root URI (http:://nvcr.io/v2 in this case):
> {code}
> GET /v2/ HTTP/1.1
> Host: nvcr.io
> User-Agent: docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f402cd kernel/4.15.0-60-generic os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.7 \(linux\))
> {code}
> To this, it receives response with a "realm" that contains no query arguments:
> {code}
> HTTP/1.1 401 Unauthorized
> Connection: close
> Content-Length: 195
> Content-Type: text/html
> Date: Wed, 29 Jan 2020 12:22:43 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth
> {code}
> Then, it composes the scope using the image ref and a hardcoded "pull" action:
> https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/registry/client/auth/session.go#L174
> (in a full accordance with this spec: https://docs.docker.com/registry/spec/auth/scope/)
> and sends the following request to https://nvcr.io/proxy_auth :
> {code}
> GET /proxy_auth?scope=repository%3Anvidia%2Ftensorflow%3Apull HTTP/1.1
> Host: nvcr.io
> User-Agent: Go-http-client/1.1
> {code}
> (Note that 'push' is absent from the scope)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)