You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by "Timothy St. Clair" <ts...@redhat.com> on 2014/08/11 17:43:40 UTC
Review Request 24555: MESOS-1169: unbundle distribute
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/
-----------------------------------------------------------
Review request for mesos, Adam B and Vinod Kone.
Bugs: MESOS-1169
https://issues.apache.org/jira/browse/MESOS-1169
Repository: mesos-git
Description
-------
Enabling unbundling for distribute python utils.
Diffs
-----
3rdparty/Makefile.am 70b45fe
configure.ac 8fb0a3a
mpi/mpiexec-mesos.in 8812ee2
src/examples/python/test-containerizer.in f71828d
src/examples/python/test-executor.in b22e7a7
src/examples/python/test-framework.in 64fb1dd
Diff: https://reviews.apache.org/r/24555/diff/
Testing
-------
./configure && make check
./configure --disable-bundled && make check
For prefixed installs require updated PYTHON_PATH.
Thanks,
Timothy St. Clair
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by "Timothy St. Clair" <ts...@redhat.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/
-----------------------------------------------------------
(Updated Aug. 14, 2014, 2:45 p.m.)
Review request for mesos, Adam B, Jie Yu, and Vinod Kone.
Changes
-------
rebase for clean apply
Bugs: MESOS-1169
https://issues.apache.org/jira/browse/MESOS-1169
Repository: mesos-git
Description
-------
Enabling unbundling for distribute python utils.
Diffs (updated)
-----
3rdparty/Makefile.am 70b45fe
configure.ac e2d8371
mpi/mpiexec-mesos.in 9aa0702
src/examples/python/test-containerizer.in 3238ccc
src/examples/python/test-executor.in 072fd7d
src/examples/python/test-framework.in ee90e55
Diff: https://reviews.apache.org/r/24555/diff/
Testing
-------
./configure && make check
./configure --disable-bundled && make check
For prefixed installs require updated PYTHON_PATH.
Thanks,
Timothy St. Clair
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/#review50549
-----------------------------------------------------------
Ship it!
Feel free to commit this.
- Vinod Kone
On Aug. 14, 2014, 3:04 a.m., Timothy St. Clair wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24555/
> -----------------------------------------------------------
>
> (Updated Aug. 14, 2014, 3:04 a.m.)
>
>
> Review request for mesos, Adam B, Jie Yu, and Vinod Kone.
>
>
> Bugs: MESOS-1169
> https://issues.apache.org/jira/browse/MESOS-1169
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enabling unbundling for distribute python utils.
>
>
> Diffs
> -----
>
> 3rdparty/Makefile.am 70b45fe
> configure.ac 8fb0a3a
> mpi/mpiexec-mesos.in 8812ee2
> src/examples/python/test-containerizer.in f71828d
> src/examples/python/test-executor.in b22e7a7
> src/examples/python/test-framework.in 64fb1dd
>
> Diff: https://reviews.apache.org/r/24555/diff/
>
>
> Testing
> -------
>
> ./configure && make check
> ./configure --disable-bundled && make check
>
> For prefixed installs require updated PYTHON_PATH.
>
>
> Thanks,
>
> Timothy St. Clair
>
>
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by "Timothy St. Clair" <ts...@redhat.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/
-----------------------------------------------------------
(Updated Aug. 14, 2014, 3:04 a.m.)
Review request for mesos, Adam B, Jie Yu, and Vinod Kone.
Changes
-------
Update per review.
Bugs: MESOS-1169
https://issues.apache.org/jira/browse/MESOS-1169
Repository: mesos-git
Description
-------
Enabling unbundling for distribute python utils.
Diffs (updated)
-----
3rdparty/Makefile.am 70b45fe
configure.ac 8fb0a3a
mpi/mpiexec-mesos.in 8812ee2
src/examples/python/test-containerizer.in f71828d
src/examples/python/test-executor.in b22e7a7
src/examples/python/test-framework.in 64fb1dd
Diff: https://reviews.apache.org/r/24555/diff/
Testing (updated)
-------
./configure && make check
./configure --disable-bundled && make check
For prefixed installs require updated PYTHON_PATH.
Thanks,
Timothy St. Clair
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by "Timothy St. Clair" <ts...@redhat.com>.
> On Aug. 13, 2014, 11:07 p.m., Vinod Kone wrote:
> > mpi/mpiexec-mesos.in, lines 21-22
> > <https://reviews.apache.org/r/24555/diff/2/?file=659530#file659530line21>
> >
> > In all these scripts, it's weird to me that we print a failed message even if the user specifically asked us to use a non-bundled egg.
> >
> > More importantly, if a bundled distribute egg exists in the build dir (perhaps from an earlier build) we favor it even when the user asked us to use the system installed version.
> >
> > My worry is that issues relating to these would be hard to diagnose down the line.
> >
> > Does this script have access to WITH_BUNDLED_DISTRIBUTE variable? If yes we should use that to determine whether to include DISTRIBUTE_EGG in the PYTHONPATH (down below where we exec). Does that make sense?
> >
> > I guess we have the same issue with protobuf egg in the earlier review.
> >
> >
For the script work I honestly think it needs a lot more work and greater in scope then just this ticket.
If it's ok with you, I would like to open some follow up tickets on this.
Right now it would be an extremely rare case for folks to reconfigure bundled/unbundled and build when stale data is around, but I completely agree the logic is obtuse and weird.
- Timothy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/#review50522
-----------------------------------------------------------
On Aug. 14, 2014, 3:04 a.m., Timothy St. Clair wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24555/
> -----------------------------------------------------------
>
> (Updated Aug. 14, 2014, 3:04 a.m.)
>
>
> Review request for mesos, Adam B, Jie Yu, and Vinod Kone.
>
>
> Bugs: MESOS-1169
> https://issues.apache.org/jira/browse/MESOS-1169
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enabling unbundling for distribute python utils.
>
>
> Diffs
> -----
>
> 3rdparty/Makefile.am 70b45fe
> configure.ac 8fb0a3a
> mpi/mpiexec-mesos.in 8812ee2
> src/examples/python/test-containerizer.in f71828d
> src/examples/python/test-executor.in b22e7a7
> src/examples/python/test-framework.in 64fb1dd
>
> Diff: https://reviews.apache.org/r/24555/diff/
>
>
> Testing
> -------
>
> ./configure && make check
> ./configure --disable-bundled && make check
>
> For prefixed installs require updated PYTHON_PATH.
>
>
> Thanks,
>
> Timothy St. Clair
>
>
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by Vinod Kone <vi...@gmail.com>.
> On Aug. 13, 2014, 11:07 p.m., Vinod Kone wrote:
> > mpi/mpiexec-mesos.in, lines 21-22
> > <https://reviews.apache.org/r/24555/diff/2/?file=659530#file659530line21>
> >
> > In all these scripts, it's weird to me that we print a failed message even if the user specifically asked us to use a non-bundled egg.
> >
> > More importantly, if a bundled distribute egg exists in the build dir (perhaps from an earlier build) we favor it even when the user asked us to use the system installed version.
> >
> > My worry is that issues relating to these would be hard to diagnose down the line.
> >
> > Does this script have access to WITH_BUNDLED_DISTRIBUTE variable? If yes we should use that to determine whether to include DISTRIBUTE_EGG in the PYTHONPATH (down below where we exec). Does that make sense?
> >
> > I guess we have the same issue with protobuf egg in the earlier review.
> >
> >
>
> Timothy St. Clair wrote:
> For the script work I honestly think it needs a lot more work and greater in scope then just this ticket.
>
> If it's ok with you, I would like to open some follow up tickets on this.
>
> Right now it would be an extremely rare case for folks to reconfigure bundled/unbundled and build when stale data is around, but I completely agree the logic is obtuse and weird.
SGTM. Please follow up with appropriate tickets. Thanks Tim.
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/#review50522
-----------------------------------------------------------
On Aug. 14, 2014, 3:04 a.m., Timothy St. Clair wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24555/
> -----------------------------------------------------------
>
> (Updated Aug. 14, 2014, 3:04 a.m.)
>
>
> Review request for mesos, Adam B, Jie Yu, and Vinod Kone.
>
>
> Bugs: MESOS-1169
> https://issues.apache.org/jira/browse/MESOS-1169
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enabling unbundling for distribute python utils.
>
>
> Diffs
> -----
>
> 3rdparty/Makefile.am 70b45fe
> configure.ac 8fb0a3a
> mpi/mpiexec-mesos.in 8812ee2
> src/examples/python/test-containerizer.in f71828d
> src/examples/python/test-executor.in b22e7a7
> src/examples/python/test-framework.in 64fb1dd
>
> Diff: https://reviews.apache.org/r/24555/diff/
>
>
> Testing
> -------
>
> ./configure && make check
> ./configure --disable-bundled && make check
>
> For prefixed installs require updated PYTHON_PATH.
>
>
> Thanks,
>
> Timothy St. Clair
>
>
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/#review50522
-----------------------------------------------------------
configure.ac
<https://reviews.apache.org/r/24555/#comment88382>
This comment needs to be updated because now the user cannot ask us to specifically use a pre-installed distribute, right? All they can do is ask us to "disable bundling" which applies to all deps.
Alternatively, I think it would be nice if users can explicitly choose to use bundled/unbundled distribute. Something along the lines of "enable_bundled_distribute" flag?
mpi/mpiexec-mesos.in
<https://reviews.apache.org/r/24555/#comment88386>
In all these scripts, it's weird to me that we print a failed message even if the user specifically asked us to use a non-bundled egg.
More importantly, if a bundled distribute egg exists in the build dir (perhaps from an earlier build) we favor it even when the user asked us to use the system installed version.
My worry is that issues relating to these would be hard to diagnose down the line.
Does this script have access to WITH_BUNDLED_DISTRIBUTE variable? If yes we should use that to determine whether to include DISTRIBUTE_EGG in the PYTHONPATH (down below where we exec). Does that make sense?
I guess we have the same issue with protobuf egg in the earlier review.
- Vinod Kone
On Aug. 13, 2014, 7:38 p.m., Timothy St. Clair wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24555/
> -----------------------------------------------------------
>
> (Updated Aug. 13, 2014, 7:38 p.m.)
>
>
> Review request for mesos, Adam B, Jie Yu, and Vinod Kone.
>
>
> Bugs: MESOS-1169
> https://issues.apache.org/jira/browse/MESOS-1169
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enabling unbundling for distribute python utils.
>
>
> Diffs
> -----
>
> 3rdparty/Makefile.am 70b45fe
> configure.ac 8fb0a3a
> mpi/mpiexec-mesos.in 8812ee2
> src/examples/python/test-containerizer.in f71828d
> src/examples/python/test-executor.in b22e7a7
> src/examples/python/test-framework.in 64fb1dd
>
> Diff: https://reviews.apache.org/r/24555/diff/
>
>
> Testing
> -------
>
> ./configure && make check
> ./configure --disable-bundled && make check
>
> For prefixed installs require updated PYTHON_PATH.
>
>
> Thanks,
>
> Timothy St. Clair
>
>
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by "Timothy St. Clair" <ts...@redhat.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/
-----------------------------------------------------------
(Updated Aug. 13, 2014, 7:38 p.m.)
Review request for mesos, Adam B, Jie Yu, and Vinod Kone.
Changes
-------
updated to remove --with-PY package, in talking with other developers the standard override is PYTHONPATH checks.
Bugs: MESOS-1169
https://issues.apache.org/jira/browse/MESOS-1169
Repository: mesos-git
Description
-------
Enabling unbundling for distribute python utils.
Diffs (updated)
-----
3rdparty/Makefile.am 70b45fe
configure.ac 8fb0a3a
mpi/mpiexec-mesos.in 8812ee2
src/examples/python/test-containerizer.in f71828d
src/examples/python/test-executor.in b22e7a7
src/examples/python/test-framework.in 64fb1dd
Diff: https://reviews.apache.org/r/24555/diff/
Testing
-------
./configure && make check
./configure --disable-bundled && make check
For prefixed installs require updated PYTHON_PATH.
Thanks,
Timothy St. Clair
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by "Timothy St. Clair" <ts...@redhat.com>.
> On Aug. 11, 2014, 6:50 p.m., Vinod Kone wrote:
> > Have you tested this with a prefix installed 'distribute'?
>
> Timothy St. Clair wrote:
> Has to be PYTHONPATH, on this one.
So after talking with others PYTHONPATH is the best override mechanics for python dependencies so simply performing the checks and removing --with-pyDEP makes the most sense here.
- Timothy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/#review50210
-----------------------------------------------------------
On Aug. 11, 2014, 3:43 p.m., Timothy St. Clair wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24555/
> -----------------------------------------------------------
>
> (Updated Aug. 11, 2014, 3:43 p.m.)
>
>
> Review request for mesos, Adam B and Vinod Kone.
>
>
> Bugs: MESOS-1169
> https://issues.apache.org/jira/browse/MESOS-1169
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enabling unbundling for distribute python utils.
>
>
> Diffs
> -----
>
> 3rdparty/Makefile.am 70b45fe
> configure.ac 8fb0a3a
> mpi/mpiexec-mesos.in 8812ee2
> src/examples/python/test-containerizer.in f71828d
> src/examples/python/test-executor.in b22e7a7
> src/examples/python/test-framework.in 64fb1dd
>
> Diff: https://reviews.apache.org/r/24555/diff/
>
>
> Testing
> -------
>
> ./configure && make check
> ./configure --disable-bundled && make check
>
> For prefixed installs require updated PYTHON_PATH.
>
>
> Thanks,
>
> Timothy St. Clair
>
>
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by "Timothy St. Clair" <ts...@redhat.com>.
> On Aug. 11, 2014, 6:50 p.m., Vinod Kone wrote:
> > Have you tested this with a prefix installed 'distribute'?
Has to be PYTHONPATH, on this one.
- Timothy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/#review50210
-----------------------------------------------------------
On Aug. 11, 2014, 3:43 p.m., Timothy St. Clair wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24555/
> -----------------------------------------------------------
>
> (Updated Aug. 11, 2014, 3:43 p.m.)
>
>
> Review request for mesos, Adam B and Vinod Kone.
>
>
> Bugs: MESOS-1169
> https://issues.apache.org/jira/browse/MESOS-1169
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enabling unbundling for distribute python utils.
>
>
> Diffs
> -----
>
> 3rdparty/Makefile.am 70b45fe
> configure.ac 8fb0a3a
> mpi/mpiexec-mesos.in 8812ee2
> src/examples/python/test-containerizer.in f71828d
> src/examples/python/test-executor.in b22e7a7
> src/examples/python/test-framework.in 64fb1dd
>
> Diff: https://reviews.apache.org/r/24555/diff/
>
>
> Testing
> -------
>
> ./configure && make check
> ./configure --disable-bundled && make check
>
> For prefixed installs require updated PYTHON_PATH.
>
>
> Thanks,
>
> Timothy St. Clair
>
>
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/#review50210
-----------------------------------------------------------
Have you tested this with a prefix installed 'distribute'?
configure.ac
<https://reviews.apache.org/r/24555/#comment87827>
How does this find a prefix installed 'distribute'?
- Vinod Kone
On Aug. 11, 2014, 3:43 p.m., Timothy St. Clair wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24555/
> -----------------------------------------------------------
>
> (Updated Aug. 11, 2014, 3:43 p.m.)
>
>
> Review request for mesos, Adam B and Vinod Kone.
>
>
> Bugs: MESOS-1169
> https://issues.apache.org/jira/browse/MESOS-1169
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enabling unbundling for distribute python utils.
>
>
> Diffs
> -----
>
> 3rdparty/Makefile.am 70b45fe
> configure.ac 8fb0a3a
> mpi/mpiexec-mesos.in 8812ee2
> src/examples/python/test-containerizer.in f71828d
> src/examples/python/test-executor.in b22e7a7
> src/examples/python/test-framework.in 64fb1dd
>
> Diff: https://reviews.apache.org/r/24555/diff/
>
>
> Testing
> -------
>
> ./configure && make check
> ./configure --disable-bundled && make check
>
> For prefixed installs require updated PYTHON_PATH.
>
>
> Thanks,
>
> Timothy St. Clair
>
>
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by Mesos ReviewBot <de...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/#review50399
-----------------------------------------------------------
Patch looks great!
Reviews applied: [24555]
All tests passed.
- Mesos ReviewBot
On Aug. 11, 2014, 3:43 p.m., Timothy St. Clair wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24555/
> -----------------------------------------------------------
>
> (Updated Aug. 11, 2014, 3:43 p.m.)
>
>
> Review request for mesos, Adam B and Vinod Kone.
>
>
> Bugs: MESOS-1169
> https://issues.apache.org/jira/browse/MESOS-1169
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enabling unbundling for distribute python utils.
>
>
> Diffs
> -----
>
> 3rdparty/Makefile.am 70b45fe
> configure.ac 8fb0a3a
> mpi/mpiexec-mesos.in 8812ee2
> src/examples/python/test-containerizer.in f71828d
> src/examples/python/test-executor.in b22e7a7
> src/examples/python/test-framework.in 64fb1dd
>
> Diff: https://reviews.apache.org/r/24555/diff/
>
>
> Testing
> -------
>
> ./configure && make check
> ./configure --disable-bundled && make check
>
> For prefixed installs require updated PYTHON_PATH.
>
>
> Thanks,
>
> Timothy St. Clair
>
>
Re: Review Request 24555: MESOS-1169: unbundle distribute
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24555/#review50234
-----------------------------------------------------------
mpi/mpiexec-mesos.in
<https://reviews.apache.org/r/24555/#comment87882>
Similar to earlier review on protobuf, can we do a python import test here to ensure the PYTHONPATH includes distribute?
- Vinod Kone
On Aug. 11, 2014, 3:43 p.m., Timothy St. Clair wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24555/
> -----------------------------------------------------------
>
> (Updated Aug. 11, 2014, 3:43 p.m.)
>
>
> Review request for mesos, Adam B and Vinod Kone.
>
>
> Bugs: MESOS-1169
> https://issues.apache.org/jira/browse/MESOS-1169
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enabling unbundling for distribute python utils.
>
>
> Diffs
> -----
>
> 3rdparty/Makefile.am 70b45fe
> configure.ac 8fb0a3a
> mpi/mpiexec-mesos.in 8812ee2
> src/examples/python/test-containerizer.in f71828d
> src/examples/python/test-executor.in b22e7a7
> src/examples/python/test-framework.in 64fb1dd
>
> Diff: https://reviews.apache.org/r/24555/diff/
>
>
> Testing
> -------
>
> ./configure && make check
> ./configure --disable-bundled && make check
>
> For prefixed installs require updated PYTHON_PATH.
>
>
> Thanks,
>
> Timothy St. Clair
>
>