Shell Environment Variables

[Hansel Chung <br...@icloud.com>]

Hi,

I had a question, I noticed that if mesos frameworks request that executors use a certain user, the env variables aren't set up for that user, but what the agent daemon was run as.

As an example:
Run mesos-slave as root:
HOME=/root
USER=root

If I have an executor to be launched as testuser, the env looks like:
HOME=/root
USER=root
instead of:
HOME=/home/testuser
USER=testuser

This is normally set by login or sudo when running as a user, but I can't seem to find a way to use these to set up a proper environment as the target user. Is there a reason we don't call getpwuid() to set a $HOME during setup of execlp() from executor.cpp? (Oh, and might as well populate $USER while we're at it)

Thanks,
-Hansel

Re: Shell Environment Variables

[Jie Yu <yu...@gmail.com>]

Another note is that Mesos task should only write to its sandbox
 ($MESOS_SANDBOX). Not sure what's the use case for getting $HOME.

On Fri, Jun 17, 2016 at 10:10 AM, Jie Yu <yu...@gmail.com> wrote:

> Hansel, I think it was definitely not intentional. I am wondering what
> happens if the user does not have a home directory, what should $HOME be?
>
> - Jie
>
> On Fri, Jun 17, 2016 at 9:08 AM, <br...@icloud.com> wrote:
>
>> Hi Haosdent,
>>
>> Yes, I had confirmed that setuid does get called correctly, and the shell
>> environment is not set up. My question is more along the lines of "is this
>> expected? is it a bug? is it intentional?"
>> If it wasn't intentional, I was thinking maybe I'll try a making a small
>> patch to see if it works out.
>>
>> I couldn't find the JIRA ticket. Happen to know what it was or anything
>> it might be referring to?
>>
>> Thanks,
>> -Hansel
>>
>> On Jun 16, 2016, at 8:00 PM, haosdent <ha...@gmail.com> wrote:
>>
>> Hi, @Hansel Chung As I checked, if --swithc_user=true (default variable
>> is true) when you launch Mesos agent, executor would call os::setuid
>> and os::setgroups before launch your tasks. But the shell environment may
>> not set. (I remember we have a ticket for this.) You could use `id` as your
>> task command to verify if user set correctly.
>>
>>
>> On Fri, Jun 17, 2016 at 9:17 AM, Hansel Chung <br...@icloud.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I had a question, I noticed that if mesos frameworks request that
>>> executors use a certain user, the env variables aren't set up for that
>>> user, but what the agent daemon was run as.
>>>
>>> As an example:
>>> Run mesos-slave as root:
>>> HOME=/root
>>> USER=root
>>>
>>> If I have an executor to be launched as testuser, the env looks like:
>>> HOME=/root
>>> USER=root
>>> instead of:
>>> HOME=/home/testuser
>>> USER=testuser
>>>
>>> This is normally set by login or sudo when running as a user, but I
>>> can't seem to find a way to use these to set up a proper environment as the
>>> target user. Is there a reason we don't call getpwuid() to set a
>>> $HOME during setup of execlp() from executor.cpp? (Oh, and might as well
>>> populate $USER while we're at it)
>>>
>>> Thanks,
>>> -Hansel
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>>
>>
>

Re: Shell Environment Variables

[Tomek Janiszewski <ja...@gmail.com>]

$HOME could be unset then homedir is looked in passwd. I had the same issue
when I was trying to map homedir with Shared Filesystem Isolator but I
thought it was issue with my setup. It will be great if this get resolved.

pt., 17.06.2016, 19:11 użytkownik Jie Yu <yu...@gmail.com> napisał:

> Hansel, I think it was definitely not intentional. I am wondering what
> happens if the user does not have a home directory, what should $HOME be?
>
> - Jie
>
> On Fri, Jun 17, 2016 at 9:08 AM, <br...@icloud.com> wrote:
>
>> Hi Haosdent,
>>
>> Yes, I had confirmed that setuid does get called correctly, and the shell
>> environment is not set up. My question is more along the lines of "is this
>> expected? is it a bug? is it intentional?"
>> If it wasn't intentional, I was thinking maybe I'll try a making a small
>> patch to see if it works out.
>>
>> I couldn't find the JIRA ticket. Happen to know what it was or anything
>> it might be referring to?
>>
>> Thanks,
>> -Hansel
>>
>> On Jun 16, 2016, at 8:00 PM, haosdent <ha...@gmail.com> wrote:
>>
>> Hi, @Hansel Chung As I checked, if --swithc_user=true (default variable
>> is true) when you launch Mesos agent, executor would call os::setuid
>> and os::setgroups before launch your tasks. But the shell environment may
>> not set. (I remember we have a ticket for this.) You could use `id` as your
>> task command to verify if user set correctly.
>>
>>
>> On Fri, Jun 17, 2016 at 9:17 AM, Hansel Chung <br...@icloud.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I had a question, I noticed that if mesos frameworks request that
>>> executors use a certain user, the env variables aren't set up for that
>>> user, but what the agent daemon was run as.
>>>
>>> As an example:
>>> Run mesos-slave as root:
>>> HOME=/root
>>> USER=root
>>>
>>> If I have an executor to be launched as testuser, the env looks like:
>>> HOME=/root
>>> USER=root
>>> instead of:
>>> HOME=/home/testuser
>>> USER=testuser
>>>
>>> This is normally set by login or sudo when running as a user, but I
>>> can't seem to find a way to use these to set up a proper environment as the
>>> target user. Is there a reason we don't call getpwuid() to set a
>>> $HOME during setup of execlp() from executor.cpp? (Oh, and might as well
>>> populate $USER while we're at it)
>>>
>>> Thanks,
>>> -Hansel
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>>
>>
>

Re: Shell Environment Variables

[Jie Yu <yu...@gmail.com>]

Hansel, I think it was definitely not intentional. I am wondering what
happens if the user does not have a home directory, what should $HOME be?

- Jie

On Fri, Jun 17, 2016 at 9:08 AM, <br...@icloud.com> wrote:

> Hi Haosdent,
>
> Yes, I had confirmed that setuid does get called correctly, and the shell
> environment is not set up. My question is more along the lines of "is this
> expected? is it a bug? is it intentional?"
> If it wasn't intentional, I was thinking maybe I'll try a making a small
> patch to see if it works out.
>
> I couldn't find the JIRA ticket. Happen to know what it was or anything it
> might be referring to?
>
> Thanks,
> -Hansel
>
> On Jun 16, 2016, at 8:00 PM, haosdent <ha...@gmail.com> wrote:
>
> Hi, @Hansel Chung As I checked, if --swithc_user=true (default variable is
> true) when you launch Mesos agent, executor would call os::setuid
> and os::setgroups before launch your tasks. But the shell environment may
> not set. (I remember we have a ticket for this.) You could use `id` as your
> task command to verify if user set correctly.
>
>
> On Fri, Jun 17, 2016 at 9:17 AM, Hansel Chung <br...@icloud.com>
> wrote:
>
>> Hi,
>>
>> I had a question, I noticed that if mesos frameworks request that
>> executors use a certain user, the env variables aren't set up for that
>> user, but what the agent daemon was run as.
>>
>> As an example:
>> Run mesos-slave as root:
>> HOME=/root
>> USER=root
>>
>> If I have an executor to be launched as testuser, the env looks like:
>> HOME=/root
>> USER=root
>> instead of:
>> HOME=/home/testuser
>> USER=testuser
>>
>> This is normally set by login or sudo when running as a user, but I can't
>> seem to find a way to use these to set up a proper environment as the
>> target user. Is there a reason we don't call getpwuid() to set a
>> $HOME during setup of execlp() from executor.cpp? (Oh, and might as well
>> populate $USER while we're at it)
>>
>> Thanks,
>> -Hansel
>>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
>
>

Re: Shell Environment Variables

[haosdent <ha...@gmail.com>]

Hi, @Hansel I think it is a known issue. I remember saw it in some mesos
issues. But I could not figure it out today. So I create a new one in
https://issues.apache.org/jira/browse/MESOS-5633 . I think the quick
workaround so far is to specify these environment variables in your task.

On Sat, Jun 18, 2016 at 12:08 AM, <br...@icloud.com> wrote:

> Hi Haosdent,
>
> Yes, I had confirmed that setuid does get called correctly, and the shell
> environment is not set up. My question is more along the lines of "is this
> expected? is it a bug? is it intentional?"
> If it wasn't intentional, I was thinking maybe I'll try a making a small
> patch to see if it works out.
>
> I couldn't find the JIRA ticket. Happen to know what it was or anything it
> might be referring to?
>
> Thanks,
> -Hansel
>
> On Jun 16, 2016, at 8:00 PM, haosdent <ha...@gmail.com> wrote:
>
> Hi, @Hansel Chung As I checked, if --swithc_user=true (default variable is
> true) when you launch Mesos agent, executor would call os::setuid
> and os::setgroups before launch your tasks. But the shell environment may
> not set. (I remember we have a ticket for this.) You could use `id` as your
> task command to verify if user set correctly.
>
>
> On Fri, Jun 17, 2016 at 9:17 AM, Hansel Chung <br...@icloud.com>
> wrote:
>
>> Hi,
>>
>> I had a question, I noticed that if mesos frameworks request that
>> executors use a certain user, the env variables aren't set up for that
>> user, but what the agent daemon was run as.
>>
>> As an example:
>> Run mesos-slave as root:
>> HOME=/root
>> USER=root
>>
>> If I have an executor to be launched as testuser, the env looks like:
>> HOME=/root
>> USER=root
>> instead of:
>> HOME=/home/testuser
>> USER=testuser
>>
>> This is normally set by login or sudo when running as a user, but I can't
>> seem to find a way to use these to set up a proper environment as the
>> target user. Is there a reason we don't call getpwuid() to set a
>> $HOME during setup of execlp() from executor.cpp? (Oh, and might as well
>> populate $USER while we're at it)
>>
>> Thanks,
>> -Hansel
>>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
>
>


-- 
Best Regards,
Haosdent Huang

Re: Shell Environment Variables

[br...@icloud.com]

Hi Haosdent,

Yes, I had confirmed that setuid does get called correctly, and the shell environment is not set up. My question is more along the lines of "is this expected? is it a bug? is it intentional?"
If it wasn't intentional, I was thinking maybe I'll try a making a small patch to see if it works out.

I couldn't find the JIRA ticket. Happen to know what it was or anything it might be referring to?

Thanks,
-Hansel

On Jun 16, 2016, at 8:00 PM, haosdent <ha...@gmail.com> wrote:

> Hi, @Hansel Chung As I checked, if --swithc_user=true (default variable is true) when you launch Mesos agent, executor would call os::setuid and os::setgroups before launch your tasks. But the shell environment may not set. (I remember we have a ticket for this.) You could use `id` as your task command to verify if user set correctly.
> 
> 
> On Fri, Jun 17, 2016 at 9:17 AM, Hansel Chung <br...@icloud.com> wrote:
> Hi,
> 
> I had a question, I noticed that if mesos frameworks request that executors use a certain user, the env variables aren't set up for that user, but what the agent daemon was run as.
> 
> As an example:
> Run mesos-slave as root:
> HOME=/root
> USER=root
> 
> If I have an executor to be launched as testuser, the env looks like:
> HOME=/root
> USER=root
> instead of:
> HOME=/home/testuser
> USER=testuser
> 
> This is normally set by login or sudo when running as a user, but I can't seem to find a way to use these to set up a proper environment as the target user. Is there a reason we don't call getpwuid() to set a $HOME during setup of execlp() from executor.cpp? (Oh, and might as well populate $USER while we're at it)
> 
> Thanks,
> -Hansel
> 
> 
> 
> -- 
> Best Regards,
> Haosdent Huang

Re: Order of URIs in CommandInfo protobuf

[Zhitao Li <zh...@gmail.com>]

Hi Robert,

I also think parallelization of fetching is important for many use cases to
reduce the time it takes to launch a task. Can we make sure the it's still
possible to parallel downloads if you file a feature request?

Also, when a task is launched, all URIs should be already fetched into
sandbox, so I'm very interested how out-of-order could break your use case.



On Mon, Jun 20, 2016 at 12:36 PM, Jie Yu <yu...@gmail.com> wrote:

> Robert, I just checked the code and the ordering is not guaranteed since
> we parallelize the download currently.
>
> This sounds like a feature request. Robert, do you want to create a
> ticket? For now, i think a startup script should be able to workaround that.
>
> On Mon, Jun 20, 2016 at 11:02 AM, Robert Lacroix <rl...@apple.com>
> wrote:
>
>> Jie, would it hurt if we would guarantee ordering of URIs? I could see
>> use cases where the order in which files are extracted matters. Protobuf
>> preserves ordering of repeated fields, so it shouldn't be a huge effort (it
>> probably already works).
>>
>>  Robert
>>
>> On Jun 17, 2016, at 7:37 PM, Jie Yu <yu...@gmail.com> wrote:
>>
>> There is no ordering assumption in the API.
>>
>> - Jie
>>
>> On Fri, Jun 17, 2016 at 10:33 AM, Wil Yegelwel <wy...@gmail.com>
>> wrote:
>>
>>> I'm curious whether there is an ordering assumption on the CommandInfo
>>> protobuf or if the order does not matter. The comment in mesos.proto, "Any
>>> URIs specified are fetched before executing the command" seems to imply
>>> that ordering does not matter. I just wanted to confirm that was the case.
>>>
>>> Thanks,
>>> Wil
>>>
>>
>>
>>
>


-- 
Cheers,

Zhitao Li

Re: Order of URIs in CommandInfo protobuf

[Jie Yu <yu...@gmail.com>]

Robert, I just checked the code and the ordering is not guaranteed since we
parallelize the download currently.

This sounds like a feature request. Robert, do you want to create a ticket?
For now, i think a startup script should be able to workaround that.

On Mon, Jun 20, 2016 at 11:02 AM, Robert Lacroix <rl...@apple.com> wrote:

> Jie, would it hurt if we would guarantee ordering of URIs? I could see use
> cases where the order in which files are extracted matters. Protobuf
> preserves ordering of repeated fields, so it shouldn't be a huge effort (it
> probably already works).
>
>  Robert
>
> On Jun 17, 2016, at 7:37 PM, Jie Yu <yu...@gmail.com> wrote:
>
> There is no ordering assumption in the API.
>
> - Jie
>
> On Fri, Jun 17, 2016 at 10:33 AM, Wil Yegelwel <wy...@gmail.com>
> wrote:
>
>> I'm curious whether there is an ordering assumption on the CommandInfo
>> protobuf or if the order does not matter. The comment in mesos.proto, "Any
>> URIs specified are fetched before executing the command" seems to imply
>> that ordering does not matter. I just wanted to confirm that was the case.
>>
>> Thanks,
>> Wil
>>
>
>
>

Re: Order of URIs in CommandInfo protobuf

[Robert Lacroix <rl...@apple.com>]

Jie, would it hurt if we would guarantee ordering of URIs? I could see use cases where the order in which files are extracted matters. Protobuf preserves ordering of repeated fields, so it shouldn't be a huge effort (it probably already works).

 Robert

> On Jun 17, 2016, at 7:37 PM, Jie Yu <yu...@gmail.com> wrote:
> 
> There is no ordering assumption in the API.
> 
> - Jie
> 
> On Fri, Jun 17, 2016 at 10:33 AM, Wil Yegelwel <wyegelwel@gmail.com <ma...@gmail.com>> wrote:
> I'm curious whether there is an ordering assumption on the CommandInfo protobuf or if the order does not matter. The comment in mesos.proto, "Any URIs specified are fetched before executing the command" seems to imply that ordering does not matter. I just wanted to confirm that was the case. 
> 
> Thanks,
> Wil
> 

Re: Order of URIs in CommandInfo protobuf

[Jie Yu <yu...@gmail.com>]

There is no ordering assumption in the API.

- Jie

On Fri, Jun 17, 2016 at 10:33 AM, Wil Yegelwel <wy...@gmail.com> wrote:

> I'm curious whether there is an ordering assumption on the CommandInfo
> protobuf or if the order does not matter. The comment in mesos.proto, "Any
> URIs specified are fetched before executing the command" seems to imply
> that ordering does not matter. I just wanted to confirm that was the case.
>
> Thanks,
> Wil
>

Order of URIs in CommandInfo protobuf

[Wil Yegelwel <wy...@gmail.com>]

I'm curious whether there is an ordering assumption on the CommandInfo
protobuf or if the order does not matter. The comment in mesos.proto, "Any
URIs specified are fetched before executing the command" seems to imply
that ordering does not matter. I just wanted to confirm that was the case.

Thanks,
Wil

Re: Shell Environment Variables

[haosdent <ha...@gmail.com>]

Hi, @Hansel Chung As I checked, if --swithc_user=true (default variable is
true) when you launch Mesos agent, executor would call os::setuid
and os::setgroups before launch your tasks. But the shell environment may
not set. (I remember we have a ticket for this.) You could use `id` as your
task command to verify if user set correctly.


On Fri, Jun 17, 2016 at 9:17 AM, Hansel Chung <br...@icloud.com> wrote:

> Hi,
>
> I had a question, I noticed that if mesos frameworks request that
> executors use a certain user, the env variables aren't set up for that
> user, but what the agent daemon was run as.
>
> As an example:
> Run mesos-slave as root:
> HOME=/root
> USER=root
>
> If I have an executor to be launched as testuser, the env looks like:
> HOME=/root
> USER=root
> instead of:
> HOME=/home/testuser
> USER=testuser
>
> This is normally set by login or sudo when running as a user, but I can't
> seem to find a way to use these to set up a proper environment as the
> target user. Is there a reason we don't call getpwuid() to set a
> $HOME during setup of execlp() from executor.cpp? (Oh, and might as well
> populate $USER while we're at it)
>
> Thanks,
> -Hansel
>



-- 
Best Regards,
Haosdent Huang