Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable

[Pierre Cheynier <me...@pierre-cheynier.net>]

> On March 22, 2016, 1:02 a.m., Joshua Cohen wrote:
> > Sorry for the delay on this. After you filed the pull request, I investigated a bit what will be required once Mesos 0.30.0 lands: https://issues.apache.org/jira/browse/AURORA-1632. I think the problem goes beyond the failure to find `sys.executable` when $PATH is not set. As even after switching back to chmod+x on the runner, the task failed further down the stack.
> > 
> > I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which should allow `sys.executable` to continue working and will allow any tasks users have running which have come to rely on Thermos setting it for them to behave as expected. The problem is, I haven't had time to figure out what we should set $PATH to yet ;) (anyone have any thoughts?).
> > 
> > I know this is probably more info than you bargained for when you opened what seemed like a simple pull request. I'm not opposed to accepting this patch (with a TODO to restore `sys.executable` when we figure out what to do about setting $PATH) if it unblocks your use case, but can you confirm that you're actually able to run the Mesos agent with `--executor_environment_variables='{}'` and still launch tasks?
> 
> Pierre Cheynier wrote:
>     Hi,
>     
>     Sorry for my delay on opening this review on apache.org.
>     Actually, I started to use Aurora and faced this issue cause my setup use `--executor_environment_variables`.
>     I first investigated in the Python default setup on CentOS, the pants/pex build system and then the differences between the vagrant box provided in the repo and mine before discovering that this is the origin of the issue.
>     
>     I admit this is a simple fix, I just tried to understand the reason of using this mechanism and found that it was changed 2 years ago.
>     In 0.27.2  it works well by using this patch (I'm now able to run something).
> 
> Joshua Cohen wrote:
>     So he's the behavior I see when I run the Mesos agent with `--executor_environment_variables='{}'`. Without this patch, as you note, tasks fail to start with a permission denied error (due to `sys.executable` being unset). After applying this patch, tasks start up, but processes fail to fork: https://www.dropbox.com/s/7zzy574xdy02ukq/Screen%20Shot%202016-03-24%20at%203.49.24%20PM.png?dl=0.
>     
>     Are you by chance running Docker tasks?
> 
> Pierre Cheynier wrote:
>     I'm running tasks on the Mesos containerizer for now.
>     Your problem is not purely a side-effect of not passing any environment variables ? (your tasks themselves maybe need PATH or LD_LIBRARY_PATH ?)
> 
> Joshua Cohen wrote:
>     It's not the task that requires PATH to be set, it's the thermos runner itself: https://github.com/apache/aurora/blob/master/src/main/python/apache/thermos/core/process.py#L397-L402
>     
>     This is why I'm confused as to how this works for you unless PATH is somehow being set elsewhere (and if PATH is set elsewhere, then `sys.executable` should work as well).

Using the Mesos Containerizer, it probably depends on the shell used and the need for the tasks itself to get a specific environment.

But I'm really surprised, cause even when I'm using your vagrant setup (patch cherry-picked on top of the master branch + vagrant up), I'm able to launch your two sample tasks (hello_world + cron_hello_world).
I can provide screenshots if needed.

The portion of code you pointed me to is supposed to enrich the environment of the subshell. I presume that if the child absolutely need the PATH/LD_LIBRARY_PATH/whatever, the operator will have to add it in --executor_environment_variable OR the user will add it by itself. This is what it's supposed to happen when Mesos 0.30 will be released, right ?


- Pierre


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45104/#review124697
-----------------------------------------------------------


On March 22, 2016, 9:50 a.m., Pierre Cheynier wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45104/
> -----------------------------------------------------------
> 
> (Updated March 22, 2016, 9:50 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> When using `--executor_environment_variables` without explicitely
> passing LD_LIBRARY_PATH, `sys.executable` returns an empty string
> resulting in a '[Errno 13] Permission denied' error for every launched
> task.
> 
> Moreover, it seems that this feature is coming in 0.30: "Executors no
> longer inherit environment variables from the agent".
> 
> This patch partially revert back 07ce21d where chmod_x method was
> removed in favor of using sys.executable.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/executor/thermos_task_runner.py 3896e3841562600379705dbf78a6f62728246348 
> 
> Diff: https://reviews.apache.org/r/45104/diff/
> 
> 
> Testing
> -------
> 
> Make Aurora executor run on CentOS7, while running mesos agent with `--executor_environment_variables` option and no excplicit $PATH set.
> 
> 
> Thanks,
> 
> Pierre Cheynier
> 
>