You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Zameer Manji <zm...@twopensource.com> on 2014/10/22 19:59:38 UTC
Re: Review Request 27009: Use virtualenv to build pants instead of
pex.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27009/#review57837
-----------------------------------------------------------
pants
<https://reviews.apache.org/r/27009/#comment98712>
Why do we need to do this?
- Zameer Manji
On Oct. 22, 2014, 10:31 a.m., Kevin Sweeney wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27009/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2014, 10:31 a.m.)
>
>
> Review request for Aurora, Bill Farner and Zameer Manji.
>
>
> Bugs: AURORA-876
> https://issues.apache.org/jira/browse/AURORA-876
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Use virtualenv to build pants instead of pex.
>
> The goal of this change is to reduce CI flakiness by using `pip install` rather than `pex` to bootstrap pants. This still reaches out to external servers (and thus has the potential to be flaky) but is at least configurable (via `--default-timeout`). In a future review we can consider mirroring to svn.apache.org and disabling PyPI lookups.
>
> This requires a change to production executor code to use `sys.executable` rather than chmod+x-ing the runner PEX. Failure to do this results in the pants virtualenv's `site-packages` shadowing the dependencies in the `pex` (fairly obviously as there is a test failure related to psutil1/2 conflicts). I'm sure I'm papering over a PEX bug here but IMO this is fine as the only reason to execute the thermos runner PEX under a different python interpreter than the executor is using (based on the hash-bang in the pex) is not a configuration we need to support IMO.
>
>
> Diffs
> -----
>
> pants 168419d7d8f5d89a0d87bfc6ac9f06c9e8e3dbc2
> src/main/python/apache/aurora/executor/thermos_task_runner.py f17910826c31c5bf754b43eb72500de639652f37
>
> Diff: https://reviews.apache.org/r/27009/diff/
>
>
> Testing
> -------
>
> git clean -fdx
> ./pants src/test/python:all
>
>
> Thanks,
>
> Kevin Sweeney
>
>
Re: Review Request 27009: Use virtualenv to build pants instead of
pex.
Posted by Kevin Sweeney <ke...@apache.org>.
> On Oct. 22, 2014, 10:59 a.m., Zameer Manji wrote:
> > pants, line 40
> > <https://reviews.apache.org/r/27009/diff/1/?file=728491#file728491line40>
> >
> > Why do we need to do this?
We don't - removed.
- Kevin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27009/#review57837
-----------------------------------------------------------
On Oct. 22, 2014, 11:05 a.m., Kevin Sweeney wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27009/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2014, 11:05 a.m.)
>
>
> Review request for Aurora, Bill Farner and Zameer Manji.
>
>
> Bugs: AURORA-876
> https://issues.apache.org/jira/browse/AURORA-876
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Use virtualenv to build pants instead of pex.
>
> The goal of this change is to reduce CI flakiness by using `pip install` rather than `pex` to bootstrap pants. This still reaches out to external servers (and thus has the potential to be flaky) but is at least configurable (via `--default-timeout`). In a future review we can consider mirroring to svn.apache.org and disabling PyPI lookups.
>
> This requires a change to production executor code to use `sys.executable` rather than chmod+x-ing the runner PEX. Failure to do this results in the pants virtualenv's `site-packages` shadowing the dependencies in the `pex` (fairly obviously as there is a test failure related to psutil1/2 conflicts). I'm sure I'm papering over a PEX bug here but IMO this is fine as the only reason to execute the thermos runner PEX under a different python interpreter than the executor is using (based on the hash-bang in the pex) is not a configuration we need to support IMO.
>
>
> Diffs
> -----
>
> pants 168419d7d8f5d89a0d87bfc6ac9f06c9e8e3dbc2
> src/main/python/apache/aurora/executor/thermos_task_runner.py f17910826c31c5bf754b43eb72500de639652f37
>
> Diff: https://reviews.apache.org/r/27009/diff/
>
>
> Testing
> -------
>
> git clean -fdx
> ./pants src/test/python:all
>
>
> Thanks,
>
> Kevin Sweeney
>
>