You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Andrei Sekretenko (Jira)" <ji...@apache.org> on 2020/01/30 10:48:00 UTC
[jira] [Created] (MESOS-10092) Cannot pull image from docker
registry which does not reply with 'scope'/'service' in WWW-Authenticate
header
Andrei Sekretenko created MESOS-10092:
-----------------------------------------
Summary: Cannot pull image from docker registry which does not reply with 'scope'/'service' in WWW-Authenticate header
Key: MESOS-10092
URL: https://issues.apache.org/jira/browse/MESOS-10092
Project: Mesos
Issue Type: Bug
Reporter: Andrei Sekretenko
Assignee: Andrei Sekretenko
This problem was encountered when trying to specify container image nvcr.io/nvidia/tensorflow:19.12-tf1-py3
When initiating Docker Registry authentication (https://docs.docker.com/registry/spec/auth/token/) with nvcr.io, Mesos URI fetcher receives 'WWW-Authenticate' header without 'service' and 'scope' params, and fails here:
https://github.com/apache/mesos/blob/1e9b121273a6d9248a78ab44798bd4c1138c31ee/src/uri/fetchers/docker.cpp#L1083
This is an example of an unsuccessful request made by Mesos:
{code}
curl -s -S -L -i --raw --http1.1 -H "Accept: application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws" -y 60 https://nvcr.io/v2/nvidia/tensorflow/manifests/19.08-py3
HTTP/1.1 401 Unauthorized
Content-Type: text/html
Date: Wed, 22 Jan 2020 19:01:57 GMT
Server: nginx/1.14.2
Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth?scope=repository:nvidia/tensorflow:pull,push"
Content-Length: 195
Connection: keep-alive
<html>
<head><title>401 Authorization Required</title></head>
<body bgcolor="white">
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
{code}
At the same time, docker is perfectly capable of pulling this image.
Note that the document "Token Authentication Specification" (https://docs.docker.com/registry/spec/auth/token/), on which the Mesos implementation is based, is vague on the issue of registries that do not provide 'scope'/'service' in WWW-Authenticate header.
What Docker does differently (at the very least, in the case of nvcr.io):
It sends the initial request not to the maniferst/blob URI, but to the repository root URI (http:://nvcr.io/v2 in this case):
{code}
GET /v2/ HTTP/1.1
Host: nvcr.io
User-Agent: docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f402cd kernel/4.15.0-60-generic os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.7 \(linux\))
{code}
To this, it receives response with a "realm" that contains no query arguments:
{code}
HTTP/1.1 401 Unauthorized
Connection: close
Content-Length: 195
Content-Type: text/html
Date: Wed, 29 Jan 2020 12:22:43 GMT
Server: nginx/1.14.2
Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth
{code}
Then, it composes the scope using the image ref and a hardcoded "pull" action:
https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/registry/client/auth/session.go#L174
(in a full accordance with this spec: https://docs.docker.com/registry/spec/auth/scope/)
and sends the following request to https://nvcr.io/proxy_auth :
{code}
GET /proxy_auth?scope=repository%3Anvidia%2Ftensorflow%3Apull HTTP/1.1
Host: nvcr.io
User-Agent: Go-http-client/1.1
{code}
(Note that 'push' is absent from the scope)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)