You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Joseph Wu <jo...@mesosphere.io> on 2018/06/20 23:55:36 UTC
Review Request 67680: Fixed archiver failure to extract hardlinks to
another directory.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67680/
-----------------------------------------------------------
Review request for mesos, Andrew Schwartzmeyer and Kapil Arya.
Bugs: MESOS-9008
https://issues.apache.org/jira/browse/MESOS-9008
Repository: mesos
Description
-------
The archiver utility may fail to extract hardlinks when the destination
target is different than the current working directory. This happens
because hardlinks are generally expressed in relative terms within
an archive, but the target should be changed to point to the extracted
location during extraction.
This updates the archiver utility to handle links correctly and adds
a regression test.
Diffs
-----
3rdparty/stout/include/stout/archiver.hpp f66da36de94f46c799cb1028b6e8c0ed26c9f3ae
3rdparty/stout/tests/archiver_tests.cpp eddda0872f02a4918cffc22ebc0da2c46b172fd9
Diff: https://reviews.apache.org/r/67680/diff/1/
Testing
-------
make check
The tarball was generated with the following:
```
echo "I'm a link target" > target
ln target hardlink
ln -s target symlink
tar -czf foo.tar.gz target hardlink symlink
```
Thanks,
Joseph Wu
Re: Review Request 67680: Fixed archiver failure to extract hardlinks
to another directory.
Posted by Kapil Arya <ka...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67680/#review205135
-----------------------------------------------------------
Ship it!
Looks good module a minor comment. Feel free to drop if it's too much.
- Kapil Arya
On June 20, 2018, 7:55 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67680/
> -----------------------------------------------------------
>
> (Updated June 20, 2018, 7:55 p.m.)
>
>
> Review request for mesos, Andrew Schwartzmeyer and Kapil Arya.
>
>
> Bugs: MESOS-9008
> https://issues.apache.org/jira/browse/MESOS-9008
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The archiver utility may fail to extract hardlinks when the destination
> target is different than the current working directory. This happens
> because hardlinks are generally expressed in relative terms within
> an archive, but the target should be changed to point to the extracted
> location during extraction.
>
> This updates the archiver utility to handle links correctly and adds
> a regression test.
>
>
> Diffs
> -----
>
> 3rdparty/stout/include/stout/archiver.hpp f66da36de94f46c799cb1028b6e8c0ed26c9f3ae
> 3rdparty/stout/tests/archiver_tests.cpp eddda0872f02a4918cffc22ebc0da2c46b172fd9
>
>
> Diff: https://reviews.apache.org/r/67680/diff/1/
>
>
> Testing
> -------
>
> make check
>
> The tarball was generated with the following:
> ```
> echo "I'm a link target" > target
> ln target hardlink
> ln -s target symlink
> tar -czf foo.tar.gz target hardlink symlink
> ```
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 67680: Fixed archiver failure to extract hardlinks
to another directory.
Posted by Andrew Schwartzmeyer <an...@schwartzmeyer.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67680/#review205194
-----------------------------------------------------------
Fix it, then Ship it!
3rdparty/stout/tests/archiver_tests.cpp
Lines 386 (patched)
<https://reviews.apache.org/r/67680/#comment288126>
This requires patching libarchive, which can/should be done separately, so just disable on Windows for now with a TODO.
I tested, this fails on Windows with _without_ the other changes too. So definitely TODO.
- Andrew Schwartzmeyer
On June 20, 2018, 4:55 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67680/
> -----------------------------------------------------------
>
> (Updated June 20, 2018, 4:55 p.m.)
>
>
> Review request for mesos, Andrew Schwartzmeyer and Kapil Arya.
>
>
> Bugs: MESOS-9008
> https://issues.apache.org/jira/browse/MESOS-9008
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The archiver utility may fail to extract hardlinks when the destination
> target is different than the current working directory. This happens
> because hardlinks are generally expressed in relative terms within
> an archive, but the target should be changed to point to the extracted
> location during extraction.
>
> This updates the archiver utility to handle links correctly and adds
> a regression test.
>
>
> Diffs
> -----
>
> 3rdparty/stout/include/stout/archiver.hpp f66da36de94f46c799cb1028b6e8c0ed26c9f3ae
> 3rdparty/stout/tests/archiver_tests.cpp eddda0872f02a4918cffc22ebc0da2c46b172fd9
>
>
> Diff: https://reviews.apache.org/r/67680/diff/1/
>
>
> Testing
> -------
>
> make check
>
> The tarball was generated with the following:
> ```
> echo "I'm a link target" > target
> ln target hardlink
> ln -s target symlink
> tar -czf foo.tar.gz target hardlink symlink
> ```
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 67680: Fixed archiver failure to extract hardlinks
to another directory.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67680/#review205153
-----------------------------------------------------------
Patch looks great!
Reviews applied: [67680]
Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose --disable-libtool-wrappers' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/docker-build.sh
- Mesos Reviewbot
On June 20, 2018, 7:55 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67680/
> -----------------------------------------------------------
>
> (Updated June 20, 2018, 7:55 p.m.)
>
>
> Review request for mesos, Andrew Schwartzmeyer and Kapil Arya.
>
>
> Bugs: MESOS-9008
> https://issues.apache.org/jira/browse/MESOS-9008
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The archiver utility may fail to extract hardlinks when the destination
> target is different than the current working directory. This happens
> because hardlinks are generally expressed in relative terms within
> an archive, but the target should be changed to point to the extracted
> location during extraction.
>
> This updates the archiver utility to handle links correctly and adds
> a regression test.
>
>
> Diffs
> -----
>
> 3rdparty/stout/include/stout/archiver.hpp f66da36de94f46c799cb1028b6e8c0ed26c9f3ae
> 3rdparty/stout/tests/archiver_tests.cpp eddda0872f02a4918cffc22ebc0da2c46b172fd9
>
>
> Diff: https://reviews.apache.org/r/67680/diff/1/
>
>
> Testing
> -------
>
> make check
>
> The tarball was generated with the following:
> ```
> echo "I'm a link target" > target
> ln target hardlink
> ln -s target symlink
> tar -czf foo.tar.gz target hardlink symlink
> ```
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 67680: Fixed archiver failure to extract hardlinks
to another directory.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67680/#review205141
-----------------------------------------------------------
FAIL: Some of the unit tests failed. Please check the relevant logs.
Reviews applied: `['67680']`
Failed command: `Start-MesosCITesting`
All the build artifacts available at: http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67680
Relevant logs:
- [stout-tests-stdout.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67680/logs/stout-tests-stdout.log):
```
[----------] 2 tests from SocketTests
[ RUN ] SocketTests.InitSocket
[ OK ] SocketTests.InitSocket (1 ms)
[ RUN ] SocketTests.IntFD
[ OK ] SocketTests.IntFD (1 ms)
[----------] 2 tests from SocketTests (5 ms total)
[----------] 2 tests from StrerrorTest
[ RUN ] StrerrorTest.ValidErrno
[ OK ] StrerrorTest.ValidErrno (0 ms)
[ RUN ] StrerrorTest.InvalidErrno
[ OK ] StrerrorTest.InvalidErrno (0 ms)
[----------] 2 tests from StrerrorTest (1 ms total)
[----------] 2 tests from OsSendfileTest
[ RUN ] OsSendfileTest.Sendfile
[ OK ] OsSendfileTest.Sendfile (1069 ms)
[ RUN ] OsSendfileTest.SendfileAsync
[ OK ] OsSendfileTest.SendfileAsync (18 ms)
[----------] 2 tests from OsSendfileTest (1089 ms total)
[----------] Global test environment tear-down
[==========] 323 tests from 49 test cases ran. (5510 ms total)
[ PASSED ] 322 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ArchiverTest.ExtractTarGzFileWithLinks
1 FAILED TEST
```
- [stout-tests-stderr.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67680/logs/stout-tests-stderr.log):
```
'invalid.command' is not recognized as an internal or external command,
operable program or batch file.
Subcommand 'subcommand' is not available
Usage: command <subcommand> [OPTIONS]
Available subcommands:
help
subcommand2
Multiple subcommands have name 'subcommand'
```
- [mesos-tests-stdout.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67680/logs/mesos-tests-stdout.log):
```
[ OK ] Endpoint/SlaveEndpointTest.NoAuthorizer/2 (119 ms)
[----------] 9 tests from Endpoint/SlaveEndpointTest (1085 ms total)
[----------] 2 tests from ContainerizerType/DefaultContainerDNSFlagTest
[ RUN ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/0
[ OK ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/0 (42 ms)
[ RUN ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/1
[ OK ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/1 (47 ms)
[----------] 2 tests from ContainerizerType/DefaultContainerDNSFlagTest (90 ms total)
[----------] 1 test from IsolationFlag/CpuIsolatorTest
[ RUN ] IsolationFlag/CpuIsolatorTest.ROOT_UserCpuUsage/0
[ OK ] IsolationFlag/CpuIsolatorTest.ROOT_UserCpuUsage/0 (799 ms)
[----------] 1 test from IsolationFlag/CpuIsolatorTest (824 ms total)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest
[ RUN ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0
[ OK ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0 (704 ms)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest (727 ms total)
[----------] Global test environment tear-down
[==========] 988 tests from 97 test cases ran. (466662 ms total)
[ PASSED ] 987 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] MesosContainerizer/DefaultExecutorTest.SigkillExecutor/0, where GetParam() = "mesos"
1 FAILED TEST
YOU HAVE 220 DISABLED TESTS
```
- [mesos-tests-stderr.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67680/logs/mesos-tests-stderr.log):
```
I0621 01:01:03.989440 12752 master.cpp:10863] Updating the state of task 85b423a0-601e-4446-bdf6-43e6b0baa9bf of framework 99d97df4-ab49-4018-ab43-9522a83c228b-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
I0621 01:01:03.989440 9372 slave.cpp:3939] Shutting down framework 99d97df4-ab49-4018-ab43-9522a83c228b-0000
I0621 01:01:03.989440 9372 slave.cpp:6660] Shutting down exeI0621 01:01:03.830482 5280 exec.cpp:162] Version: 1.7.0
I0621 01:01:03.855453 11344 exec.cpp:236] Executor registered on agent 99d97df4-ab49-4018-ab43-9522a83c228b-S0
I0621 01:01:03.859453 2560 executor.cpp:178] Received SUBSCRIBED event
I0621 01:01:03.863478 2560 executor.cpp:182] Subscribed executor on windows-02.enofukwu14ruplxn0gs3yzmsgf.xx.internal.cloudapp.net
I0621 01:01:03.863478 2560 executor.cpp:178] Received LAUNCH event
I0621 01:01:03.868466 2560 executor.cpp:665] Starting task 85b423a0-601e-4446-bdf6-43e6b0baa9bf
I0621 01:01:03.948482 2560 executor.cpp:485] Running 'D:\DCOS\mesos\src\mesos-containerizer.exe launch <POSSIBLY-SENSITIVE-DATA>'
I0621 01:01:03.961462 2560 executor.cpp:678] Forked command at 10096
I0621 01:01:03.992444 13716 exec.cpp:445] Executor asked to shutdown
I0621 01:01:03.993446 2560 executor.cpp:178] Received SHUTDOWN event
I0621 01:01:03.993446 2560 executor.cpp:781] Shutting down
I0621 01:01:03.993446 2560 executor.cpp:894] Sending SIGTERM to process tree at pid 10cutor '85b423a0-601e-4446-bdf6-43e6b0baa9bf' of framework 99d97df4-ab49-4018-ab43-9522a83c228b-0000 at executor(1)@192.10.1.6:62893
I0621 01:01:03.991446 9372 slave.cpp:931] Agent terminating
W0621 01:01:03.992444 9372 slave.cpp:3935] Ignoring shutdown framework 99d97df4-ab49-4018-ab43-9522a83c228b-0000 because it is terminating
I0621 01:01:03.992444 12752 master.cpp:10962] Removing task 85b423a0-601e-4446-bdf6-43e6b0baa9bf with resources cpus(allocated: *):4; mem(allocated: *):2048; disk(allocated: *):1024; ports(allocated: *):[31000-32000] of framework 99d97df4-ab49-4018-ab43-9522a83c228b-0000 on agent 99d97df4-ab49-4018-ab43-9522a83c228b-S0 at slave(449)@192.10.1.6:62872 (windows-02.enofukwu14ruplxn0gs3yzmsgf.xx.internal.cloudapp.net)
I0621 01:01:03.995443 12752 master.cpp:1293] Agent 99d97df4-ab49-4018-ab43-9522a83c228b-S0 at slave(449)@192.10.1.6:62872 (windows-02.enofukwu14ruplxn0gs3yzmsgf.xx.internal.cloudapp.net) disconnected
I0621 01:01:03.995443 12752 master.cpp:3303] Disconnecting agent 99d97df4-ab49-4018-ab43-9522a83c228b-S0 at slave(449)@192.10.1.6:62872 (windows-02.enofukwu14ruplxn0gs3yzmsgf.xx.internal.cloudapp.net)
I0621 01:01:03.996443 12752 master.cpp:3322] Deactivating agent 99d97df4-ab49-4018-ab43-9522a83c228b-S0 at slave(449)@192.10.1.6:62872 (windows-02.enofukwu14ruplxn0gs3yzmsgf.xx.internal.cloudapp.net)
I0621 01:01:03.996443 4220 hierarchical.cpp:344] Removed framework 99d97df4-ab49-4018-ab43-9522a83c228b-0000
I0621 01:01:03.997443 4220 hierarchical.cpp:766] Agent 99d97df4-ab49-4018-ab43-9522a83c228b-S0 deactivated
I0621 01:01:03.997443 9372 containerizer.cpp:2407] Destroying container f0ac6b70-01eb-4a4a-bbf9-14ea0d19ba49 in RUNNING state
I0621 01:01:03.997443 9372 containerizer.cpp:3021] Transitioning the state of container f0ac6b70-01eb-4a4a-bbf9-14ea0d19ba49 from RUNNING to DESTROYING
I0621 01:01:03.998453 9372 launcher.cpp:155] Asked to destroy container f0ac6b70-01eb-4a4a-bbf9-14ea0d19ba49
I0621 01:01:04.028475 12752 containerizer.cpp:2860] Container f0ac6b70-01eb-4a4a-bbf9-14ea0d19ba49 has exited
I0621 01:01:04.059453 3180 master.cpp:1135] Master terminating
I0621 01:01:04.061455 6272 hierarchical.cpp:609] Removed agent 99d97df4-ab49-4018-ab43-9522a83c228b-S0
I0621 01:01:04.405475 11304 process.cpp:940] Stopped the socket accept loop
```
- Mesos Reviewbot Windows
On June 20, 2018, 11:55 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67680/
> -----------------------------------------------------------
>
> (Updated June 20, 2018, 11:55 p.m.)
>
>
> Review request for mesos, Andrew Schwartzmeyer and Kapil Arya.
>
>
> Bugs: MESOS-9008
> https://issues.apache.org/jira/browse/MESOS-9008
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The archiver utility may fail to extract hardlinks when the destination
> target is different than the current working directory. This happens
> because hardlinks are generally expressed in relative terms within
> an archive, but the target should be changed to point to the extracted
> location during extraction.
>
> This updates the archiver utility to handle links correctly and adds
> a regression test.
>
>
> Diffs
> -----
>
> 3rdparty/stout/include/stout/archiver.hpp f66da36de94f46c799cb1028b6e8c0ed26c9f3ae
> 3rdparty/stout/tests/archiver_tests.cpp eddda0872f02a4918cffc22ebc0da2c46b172fd9
>
>
> Diff: https://reviews.apache.org/r/67680/diff/1/
>
>
> Testing
> -------
>
> make check
>
> The tarball was generated with the following:
> ```
> echo "I'm a link target" > target
> ln target hardlink
> ln -s target symlink
> tar -czf foo.tar.gz target hardlink symlink
> ```
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 67680: Fixed archiver failure to extract hardlinks
to another directory.
Posted by Kapil Arya <ka...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67680/#review205134
-----------------------------------------------------------
3rdparty/stout/tests/archiver_tests.cpp
Lines 388-389 (patched)
<https://reviews.apache.org/r/67680/#comment288098>
Should we add a short comment on how the contents were generated? Basically, copy-paste commands from the Testing Done section of this RR?
- Kapil Arya
On June 20, 2018, 7:55 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67680/
> -----------------------------------------------------------
>
> (Updated June 20, 2018, 7:55 p.m.)
>
>
> Review request for mesos, Andrew Schwartzmeyer and Kapil Arya.
>
>
> Bugs: MESOS-9008
> https://issues.apache.org/jira/browse/MESOS-9008
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The archiver utility may fail to extract hardlinks when the destination
> target is different than the current working directory. This happens
> because hardlinks are generally expressed in relative terms within
> an archive, but the target should be changed to point to the extracted
> location during extraction.
>
> This updates the archiver utility to handle links correctly and adds
> a regression test.
>
>
> Diffs
> -----
>
> 3rdparty/stout/include/stout/archiver.hpp f66da36de94f46c799cb1028b6e8c0ed26c9f3ae
> 3rdparty/stout/tests/archiver_tests.cpp eddda0872f02a4918cffc22ebc0da2c46b172fd9
>
>
> Diff: https://reviews.apache.org/r/67680/diff/1/
>
>
> Testing
> -------
>
> make check
>
> The tarball was generated with the following:
> ```
> echo "I'm a link target" > target
> ln target hardlink
> ln -s target symlink
> tar -czf foo.tar.gz target hardlink symlink
> ```
>
>
> Thanks,
>
> Joseph Wu
>
>