You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@mesos.apache.org by Till Toenshoff via Review Board <no...@reviews.apache.org> on 2018/10/18 23:41:21 UTC

Re: Review Request 69075: Updated docker image fetcher to enforce HTTP1.0.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69075/
-----------------------------------------------------------

(Updated Oct. 18, 2018, 11:41 p.m.)

Review request for mesos, Alexander Rukletsov, Armand Grillet, Jie Yu, and James Peach.

Changes
-------

updated to support centos 6 and 7

Summary (updated)
-----------------

Updated docker image fetcher to enforce HTTP1.0.

Bugs: MESOS-8907
https://issues.apache.org/jira/browse/MESOS-8907

Repository: mesos

Description (updated)
-------

Modifies the 'curl' invocation that is returning an http::Response,
locking it into HTTP 1.0. Our current HTTP parser is unable to process
HTTP 2 responses.

HTTP 1.0 was chosen instead of HTTP 1.1 as some supported linux
distributions still come with curl 7.19 which does not support the
'--http1.1' flag.

With the advent of curl 7.47, HTTPS connections are being enforced
towards HTTP 2 rather aggressively. As a result, our image fetcher
fails when recent curl versions are being used for pulling images from
a registry that supports HTTP 2.

For allowing all the benefits of HTTP 2 where possible, we do not
adapt any 'curl' invocations that do not attempt to parse headers.

Diffs (updated)
-----

src/uri/fetchers/docker.cpp 6b1277f822dcf80d44daa8133b3ecc8c9a34ef07

Diff: https://reviews.apache.org/r/69075/diff/2/

Changes: https://reviews.apache.org/r/69075/diff/1-2/

Testing (updated)
-------

`make check`
`sudo ./bin/mesos-tests.sh --gtest_filter="ImageAlpine/ProvisionerDockerTest.ROOT_INTERNET_CURL_SimpleCommand/2"` on a system with curl 7.59.0 installed.

Thanks,

Till Toenshoff

Re: Review Request 69075: Updated docker image fetcher to enforce HTTP 1.x.

Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69075/#review209854
-----------------------------------------------------------



PASS: Mesos patch 69075 was successfully built and tested.

Reviews applied: `['69075']`

All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2505/mesos-review-69075

- Mesos Reviewbot Windows


On Oct. 22, 2018, 7:22 p.m., Till Toenshoff wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69075/
> -----------------------------------------------------------
> 
> (Updated Oct. 22, 2018, 7:22 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Armand Grillet, Jie Yu, and James Peach.
> 
> 
> Bugs: MESOS-8907
>     https://issues.apache.org/jira/browse/MESOS-8907
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Modifies the 'curl' invocation that is returning an http::Response,
> locking it into HTTP 1.x. Our current HTTP parser is unable to process
> HTTP 2 responses.
> 
> With the advent of curl 7.47, HTTPS connections are being enforced
> towards HTTP 2 rather aggressively. As a result, our image fetcher
> fails when recent curl versions are being used for pulling images from
> a registry that supports HTTP 2.
> 
> HTTP 1.1 is chosen as long as the underlying curl supports the
> '--http1.1' flag. If curl does not support that flag, fall back to
> using HTTP 1.0.
> 
> For allowing all the benefits of HTTP 2 where possible, we do not
> adapt any 'curl' invocations that do not attempt to parse headers.
> 
> 
> Diffs
> -----
> 
>   src/uri/fetchers/docker.cpp 6b1277f822dcf80d44daa8133b3ecc8c9a34ef07 
> 
> 
> Diff: https://reviews.apache.org/r/69075/diff/3/
> 
> 
> Testing
> -------
> 
> `make check`
> `sudo ./bin/mesos-tests.sh --gtest_filter="ImageAlpine/ProvisionerDockerTest.ROOT_INTERNET_CURL_SimpleCommand/2"` on a system with curl 7.59.0 installed.
> 
> 
> Thanks,
> 
> Till Toenshoff
> 
>

Re: Review Request 69075: Updated docker image fetcher to enforce HTTP 1.1.

Posted by James Peach <jp...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69075/#review209873
-----------------------------------------------------------


Fix it, then Ship it!





src/uri/fetchers/docker.cpp
Lines 124 (patched)
<https://reviews.apache.org/r/69075/#comment294488>

    I think we can improve this comment ..
    
    > the HTTP 1.1 locking
    
    ??
    
    > therefor
    
    therefore
    
    Maybe:
    ```
    Older curl versions do not support the HTTP.1.1 flag, but these versions are also old enough to not default to HTTP/2.
    ```


- James Peach


On Oct. 22, 2018, 4:22 p.m., Till Toenshoff wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69075/
> -----------------------------------------------------------
> 
> (Updated Oct. 22, 2018, 4:22 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Armand Grillet, Jie Yu, and James Peach.
> 
> 
> Bugs: MESOS-8907
>     https://issues.apache.org/jira/browse/MESOS-8907
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Modifies the 'curl' invocation that is returning an http::Response,
> locking it into HTTP 1.1. Our current HTTP parser is unable to process
> HTTP 2 responses.
> 
> With the advent of curl 7.47, HTTPS connections are being enforced
> towards HTTP 2 rather aggressively. As a result, our image fetcher
> fails when recent curl versions are being used for pulling images from
> a registry that supports HTTP 2.
> 
> HTTP 1.1 is chosen as long as the underlying curl supports the
> '--http1.1' flag. If curl is old enough to not support that flag, we
> can deduct that it will not enforce HTTP 2 and therefor need no
> further actions.
> 
> For allowing all the benefits of HTTP 2 where possible, we do not
> adapt any 'curl' invocations that do not attempt to parse headers.
> 
> 
> Diffs
> -----
> 
>   src/uri/fetchers/docker.cpp 6b1277f822dcf80d44daa8133b3ecc8c9a34ef07 
> 
> 
> Diff: https://reviews.apache.org/r/69075/diff/4/
> 
> 
> Testing
> -------
> 
> `make check`
> `sudo ./bin/mesos-tests.sh --gtest_filter="ImageAlpine/ProvisionerDockerTest.ROOT_INTERNET_CURL_SimpleCommand/2"` on a system with curl 7.59.0 installed.
> 
> 
> Thanks,
> 
> Till Toenshoff
> 
>

Re: Review Request 69075: Updated docker image fetcher to enforce HTTP 1.1 where needed.

Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69075/#review209887
-----------------------------------------------------------



FAIL: Some of the unit tests failed. Please check the relevant logs.

Reviews applied: `['69075']`

Failed command: `Start-MesosCITesting`

All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2509/mesos-review-69075

Relevant logs:

- [mesos-tests.log](http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2509/mesos-review-69075/logs/mesos-tests.log):

```
I1022 17:21:30.089454 21200 executor.cpp:184] Received SHUTDOWN event
I1022 17:21:30.089454 21200 executor.cpp:805] Shutting down
I1022 17:21:30.089454 21200 executor.cpp:918] Sending SIGTERM to process tree at pid 276task 28bfbb95-cf0c-4807-8fd7-fa2b82ee7881 with resources cpus(allocated: *):4; mem(allocated: *):2048; disk(allocated: *):1024; ports(allocated: *):[31000-32000] of framework 3c466bf9-2352-43d0-84ec-6ed9760f3391-0000 on agent 3c466bf9-2352-43d0-84ec-6ed9760f3391-S0 at slave(461)@192.10.1.8:49162 (windows-05.aa0q4n2kgcyefckmv0xukjvy4f.xx.internal.cloudapp.net)
I1022 17:21:30.089454 30408 slave.cpp:915] Agent terminating
W1022 17:21:30.089454 30408 slave.cpp:3923] Ignoring shutdown framework 3c466bf9-2352-43d0-84ec-6ed9760f3391-0000 because it is terminating
I1022 17:21:30.091459 25328 master.cpp:1266] Agent 3c466bf9-2352-43d0-84ec-6ed9760f3391-S0 at slave(461)@192.10.1.8:49162 (windows-05.aa0q4n2kgcyefckmv0xukjvy4f.xx.internal.cloudapp.net) disconnected
I1022 17:21:30.091459 25328 master.cpp:3282] Disconnecting agent 3c466bf9-2352-43d0-84ec-6ed9760f3391-S0 at slave(461)@192.10.1.8:49162 (windows-05.aa0q4n2kgcyefckmv0xukjvy4f.xx.internal.cloudapp.net)
I1022 17:21:30.091459 25328 master.cpp:3301] Deactivating agent 3c466bf9-2352-43d0-84ec-6ed9760f3391-S0 at slave(461)@192.10.1.8:49162 (windows-05.aa0q4n2kgcyefckmv0xukjvy4f.xx.internal.cloudapp.net)
I1022 17:21:30.092567 27548 hierarchical.cpp:359] Removed framework 3c466bf9-2352-43d0-84ec-6ed9760f3391-0000
I1022 17:21:30.092567 27548 hierarchical.cpp:803] Agent 3c466bf9-2352-43d0-84ec-6ed9760f3391-S0 deactivated
I1022 17:21:30.093459 29184 containerizer.cpp:2455] Destroying container a811e304-31d7-4c6d-a9b6-a9698407afa1 in RUNNING state
I1022 17:21:30.094449 29184 containerizer.cpp:3122] Transitioning the state of container a811e304-31d7-4c6d-a9b6-a9698407afa1 from RUNNING to DESTROYING
I1022 17:21:30.094449 29184 launcher.cpp:166] Asked to destroy container a811e304-31d7-4c6d-a9b6-a9698407afa1
I1022 17:21:30.191465 25328 containerizer.cpp:2961] Container a811e304-31d7-4c6d-a9b6-a9698407afa1 has exited
I1022 17:21:30.221736 24116 master.cpp:1108] Master terminating
I1022 17:21:30.223459 30628 hierarchical.cpp:645] Removed agent 3c466bf9-2352-43d0-84ec-6ed9760f3391-S[       OK ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0 (684 ms)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest (704 ms total)

[----------] Global test environment tear-down
[==========] 1052 tests from 103 test cases ran. (543332 ms total)
[  PASSED  ] 1051 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DockerFetcherPluginTest.INTERNET_CURL_InvokeFetchByName

 1 FAILED TEST
  YOU HAVE 232 DISABLED TESTS

0
I1022 17:21:30.594519 30068 process.cpp:926] Stopped the socket accept loop
```

- Mesos Reviewbot Windows


On Oct. 22, 2018, 4:22 p.m., Till Toenshoff wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69075/
> -----------------------------------------------------------
> 
> (Updated Oct. 22, 2018, 4:22 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Armand Grillet, Jie Yu, and James Peach.
> 
> 
> Bugs: MESOS-8907
>     https://issues.apache.org/jira/browse/MESOS-8907
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Modifies the 'curl' invocation that is returning an http::Response,
> locking it into HTTP 1.1. Our current HTTP parser is unable to process
> HTTP 2 responses.
> 
> With the advent of curl 7.47, HTTPS connections are being enforced
> towards HTTP 2 rather aggressively. As a result, our image fetcher
> fails when recent curl versions are being used for pulling images from
> a registry that supports HTTP 2.
> 
> HTTP 1.1 is chosen as long as the underlying curl supports the
> '--http1.1' flag. If curl is old enough to not support that flag, we
> can deduct that it will not enforce HTTP 2 and therefore need no
> further actions.
> 
> For allowing all the benefits of HTTP 2 where possible, we do not
> adapt any 'curl' invocations that do not attempt to parse headers.
> 
> 
> Diffs
> -----
> 
>   src/uri/fetchers/docker.cpp 6b1277f822dcf80d44daa8133b3ecc8c9a34ef07 
> 
> 
> Diff: https://reviews.apache.org/r/69075/diff/5/
> 
> 
> Testing
> -------
> 
> `make check`
> `sudo ./bin/mesos-tests.sh --gtest_filter="ImageAlpine/ProvisionerDockerTest.ROOT_INTERNET_CURL_SimpleCommand/2"` on a system with curl 7.59.0 installed.
> 
> 
> Thanks,
> 
> Till Toenshoff
> 
>

Re: Review Request 69075: Updated docker image fetcher to enforce HTTP 1.1.

Posted by Till Toenshoff via Review Board <no...@reviews.apache.org>.

(Updated Oct. 22, 2018, 4:22 p.m.)

Review request for mesos, Alexander Rukletsov, Armand Grillet, Jie Yu, and James Peach.

Changes
-------

Updated again to address recent comment by James - thanks a bunch!

Summary (updated)
-----------------

Updated docker image fetcher to enforce HTTP 1.1.

Bugs: MESOS-8907
https://issues.apache.org/jira/browse/MESOS-8907

Repository: mesos

Description (updated)
-------

Modifies the 'curl' invocation that is returning an http::Response,
locking it into HTTP 1.1. Our current HTTP parser is unable to process
HTTP 2 responses.

HTTP 1.1 is chosen as long as the underlying curl supports the
'--http1.1' flag. If curl is old enough to not support that flag, we
can deduct that it will not enforce HTTP 2 and therefor need no
further actions.

For allowing all the benefits of HTTP 2 where possible, we do not
adapt any 'curl' invocations that do not attempt to parse headers.

Diffs (updated)
-----

src/uri/fetchers/docker.cpp 6b1277f822dcf80d44daa8133b3ecc8c9a34ef07

Diff: https://reviews.apache.org/r/69075/diff/4/

Changes: https://reviews.apache.org/r/69075/diff/3-4/

Testing
-------

`make check`
`sudo ./bin/mesos-tests.sh --gtest_filter="ImageAlpine/ProvisionerDockerTest.ROOT_INTERNET_CURL_SimpleCommand/2"` on a system with curl 7.59.0 installed.

Thanks,

Till Toenshoff

Re: Review Request 69075: Updated docker image fetcher to enforce HTTP 1.x.

Posted by Till Toenshoff via Review Board <no...@reviews.apache.org>.


> On Oct. 22, 2018, 4:03 p.m., James Peach wrote:
> > src/uri/fetchers/docker.cpp
> > Lines 123 (patched)
> > <https://reviews.apache.org/r/69075/diff/3/?file=2101778#file2101778line123>
> >
> >     If curl is so old that it doesn't accept the `--http1.1` flags, then it's not going to use HTTP/2 by default. In this case, there's no point forcing HTTP/1.0.

That way we could avoid forcing 1.0 when 1.1 was actually working (just not offered as a flag) - very good point - thank you!


- Till


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69075/#review209868
-----------------------------------------------------------


On Oct. 22, 2018, 11:22 a.m., Till Toenshoff wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69075/
> -----------------------------------------------------------
> 
> (Updated Oct. 22, 2018, 11:22 a.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Armand Grillet, Jie Yu, and James Peach.
> 
> 
> Bugs: MESOS-8907
>     https://issues.apache.org/jira/browse/MESOS-8907
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Modifies the 'curl' invocation that is returning an http::Response,
> locking it into HTTP 1.x. Our current HTTP parser is unable to process
> HTTP 2 responses.
> 
> With the advent of curl 7.47, HTTPS connections are being enforced
> towards HTTP 2 rather aggressively. As a result, our image fetcher
> fails when recent curl versions are being used for pulling images from
> a registry that supports HTTP 2.
> 
> HTTP 1.1 is chosen as long as the underlying curl supports the
> '--http1.1' flag. If curl does not support that flag, fall back to
> using HTTP 1.0.
> 
> For allowing all the benefits of HTTP 2 where possible, we do not
> adapt any 'curl' invocations that do not attempt to parse headers.
> 
> 
> Diffs
> -----
> 
>   src/uri/fetchers/docker.cpp 6b1277f822dcf80d44daa8133b3ecc8c9a34ef07 
> 
> 
> Diff: https://reviews.apache.org/r/69075/diff/3/
> 
> 
> Testing
> -------
> 
> `make check`
> `sudo ./bin/mesos-tests.sh --gtest_filter="ImageAlpine/ProvisionerDockerTest.ROOT_INTERNET_CURL_SimpleCommand/2"` on a system with curl 7.59.0 installed.
> 
> 
> Thanks,
> 
> Till Toenshoff
> 
>

Re: Review Request 69075: Updated docker image fetcher to enforce HTTP 1.x.

Posted by James Peach <jp...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69075/#review209868
-----------------------------------------------------------




src/uri/fetchers/docker.cpp
Lines 123 (patched)
<https://reviews.apache.org/r/69075/#comment294477>

    If curl is so old that it doesn't accept the `--http1.1` flags, then it's not going to use HTTP/2 by default. In this case, there's no point forcing HTTP/1.0.


- James Peach


On Oct. 22, 2018, 11:22 a.m., Till Toenshoff wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69075/
> -----------------------------------------------------------
> 
> (Updated Oct. 22, 2018, 11:22 a.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Armand Grillet, Jie Yu, and James Peach.
> 
> 
> Bugs: MESOS-8907
>     https://issues.apache.org/jira/browse/MESOS-8907
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Modifies the 'curl' invocation that is returning an http::Response,
> locking it into HTTP 1.x. Our current HTTP parser is unable to process
> HTTP 2 responses.
> 
> With the advent of curl 7.47, HTTPS connections are being enforced
> towards HTTP 2 rather aggressively. As a result, our image fetcher
> fails when recent curl versions are being used for pulling images from
> a registry that supports HTTP 2.
> 
> HTTP 1.1 is chosen as long as the underlying curl supports the
> '--http1.1' flag. If curl does not support that flag, fall back to
> using HTTP 1.0.
> 
> For allowing all the benefits of HTTP 2 where possible, we do not
> adapt any 'curl' invocations that do not attempt to parse headers.
> 
> 
> Diffs
> -----
> 
>   src/uri/fetchers/docker.cpp 6b1277f822dcf80d44daa8133b3ecc8c9a34ef07 
> 
> 
> Diff: https://reviews.apache.org/r/69075/diff/3/
> 
> 
> Testing
> -------
> 
> `make check`
> `sudo ./bin/mesos-tests.sh --gtest_filter="ImageAlpine/ProvisionerDockerTest.ROOT_INTERNET_CURL_SimpleCommand/2"` on a system with curl 7.59.0 installed.
> 
> 
> Thanks,
> 
> Till Toenshoff
> 
>

Re: Review Request 69075: Updated docker image fetcher to enforce HTTP 1.x.

Posted by Armand Grillet <ag...@mesosphere.io>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69075/#review209845
-----------------------------------------------------------


Ship it!




Ship It!

- Armand Grillet


On Oct. 22, 2018, 1:22 p.m., Till Toenshoff wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69075/
> -----------------------------------------------------------
> 
> (Updated Oct. 22, 2018, 1:22 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Armand Grillet, Jie Yu, and James Peach.
> 
> 
> Bugs: MESOS-8907
>     https://issues.apache.org/jira/browse/MESOS-8907
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Modifies the 'curl' invocation that is returning an http::Response,
> locking it into HTTP 1.x. Our current HTTP parser is unable to process
> HTTP 2 responses.
> 
> With the advent of curl 7.47, HTTPS connections are being enforced
> towards HTTP 2 rather aggressively. As a result, our image fetcher
> fails when recent curl versions are being used for pulling images from
> a registry that supports HTTP 2.
> 
> HTTP 1.1 is chosen as long as the underlying curl supports the
> '--http1.1' flag. If curl does not support that flag, fall back to
> using HTTP 1.0.
> 
> For allowing all the benefits of HTTP 2 where possible, we do not
> adapt any 'curl' invocations that do not attempt to parse headers.
> 
> 
> Diffs
> -----
> 
>   src/uri/fetchers/docker.cpp 6b1277f822dcf80d44daa8133b3ecc8c9a34ef07 
> 
> 
> Diff: https://reviews.apache.org/r/69075/diff/3/
> 
> 
> Testing
> -------
> 
> `make check`
> `sudo ./bin/mesos-tests.sh --gtest_filter="ImageAlpine/ProvisionerDockerTest.ROOT_INTERNET_CURL_SimpleCommand/2"` on a system with curl 7.59.0 installed.
> 
> 
> Thanks,
> 
> Till Toenshoff
> 
>

Re: Review Request 69075: Updated docker image fetcher to enforce HTTP 1.x.

Posted by Till Toenshoff via Review Board <no...@reviews.apache.org>.

(Updated Oct. 22, 2018, 11:22 a.m.)

Review request for mesos, Alexander Rukletsov, Armand Grillet, Jie Yu, and James Peach.

Changes
-------

Make HTTP 1.1 default but allow for fallback on HTTP 1.0 when curl does not support otherwise.

Summary (updated)
-----------------

Updated docker image fetcher to enforce HTTP 1.x.

Bugs: MESOS-8907
https://issues.apache.org/jira/browse/MESOS-8907

Repository: mesos

Description (updated)
-------

Modifies the 'curl' invocation that is returning an http::Response,
locking it into HTTP 1.x. Our current HTTP parser is unable to process
HTTP 2 responses.

HTTP 1.1 is chosen as long as the underlying curl supports the
'--http1.1' flag. If curl does not support that flag, fall back to
using HTTP 1.0.

For allowing all the benefits of HTTP 2 where possible, we do not
adapt any 'curl' invocations that do not attempt to parse headers.

Diffs (updated)
-----

src/uri/fetchers/docker.cpp 6b1277f822dcf80d44daa8133b3ecc8c9a34ef07

Diff: https://reviews.apache.org/r/69075/diff/3/

Changes: https://reviews.apache.org/r/69075/diff/2-3/

Testing
-------

`make check`
`sudo ./bin/mesos-tests.sh --gtest_filter="ImageAlpine/ProvisionerDockerTest.ROOT_INTERNET_CURL_SimpleCommand/2"` on a system with curl 7.59.0 installed.

Thanks,

Till Toenshoff