You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Kiril Menshikov <km...@gmail.com> on 2016/12/29 17:07:09 UTC
Too many files
Hi,
I have executor which run java programs. One executor execute around 800
tasks. Last ~100 failed with "Too many open files". I increased nofile and
nproc limits. During my debug I could not say that problem in in the tasks.
But some times linux reach limits. I see some boxes are good with out 'Too
many open files errors'. But some has such errors.
I run executor through mesos-containerizer and isolation is posix/cpu,posix/
mem.
Can some one explain why this happens? Is it better to create separate
executor for each task? Tasks have common code but has different commands.
Any help are welcomed.
mesos-containerizer launch --command={"shell":true,"value":"java -cp
executor-all-1.0.jar com.stone.mesos.MesosTestExecutor"} --environment={"
LIBPROCESS_IP":"10.10.10.10","LIBPROCESS_PORT":"0","MESOS_AGENT_ENDPOINT":"
10.10.10.10:5051","MESOS_CHECKPOINT":"1","MESOS
_DIRECTORY":"\/var\/lib\/mesos\/slaves\/7e30a916-1296-4f47-813a-0972030b6907-S14\/frameworks\/7e30a916-1296-4f47-813a-0972030b6907-0020\/executors\/client_b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381\/runs\/ee8857e2-ac19-4f07-810d-c2e71fbf522e","
MESOS_EXECUTOR_ID":"client_b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381","
MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD":"5secs","MESOS
_FRAMEWORK_ID":"7e30a916-1296-4f47-813a-0972030b6907-0020","MESOS
_HTTP_COMMAND_EXECUTOR":"0","MESOS_NATIVE_JAVA_LIBRARY":"\/usr\/local\/lib\/
libmesos-1.1.0.so","MESOS_NATIVE_LIBRARY":"\/usr\/local\/lib\/libmesos-
1.1.0.so","MESOS_RECOVERY_TIMEOUT":"15mins","MESOS
_SANDBOX":"\/var\/lib\/mesos\/slaves\/7e30a916-1296-4f47-813a-0972030b6907-S14\/frameworks\/7e30a916-1296-4f47-813a-0972030b6907-0020\/executors\/client_b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381\/runs\/ee8857e2-ac19-4f07-810d-c2e71fbf522e","
MESOS_SLAVE_ID":"7e30a916-1296-4f47-813a-0972030b6907-S14","MESOS_SLAVE_PID
":"slave(1)@10.10.10.10:5051","MESOS_SUBSCRIPTION_BACKOFF
_MAX":"2secs","PATH":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin
:\/usr\/bin:\/sbin:\/bin"} --help=false --pipe_read=12 --pipe_write=13
--pre_exec_commands=[]
--runtime_directory=/var/run/mesos/containers/ee8857e2-ac19-4f07-810d-c2e71fbf522e
--unshare_namespace_mnt=false --user=ec2-user --working_directory=/var/lib/
mesos
/slaves/7e30a916-1296-4f47-813a-0972030b6907-S14/frameworks/7e30a916-1296-4f47-813a-0972030b6907-0020/executors/client_b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381/runs/ee8857e2-ac19-4f07-810d-c2e71fbf522e
Thanks,
-Kiril
Re: Too many files
Posted by haosdent <ha...@gmail.com>.
No. It means executor/task only could open 4000 files exclude child
processes.
On Fri, Dec 30, 2016 at 2:32 AM, Kiril Menshikov <km...@gmail.com>
wrote:
> It’s 4000. Does this mean that sum of all child open files should not
> exceed this value? Because I see that task limit is also 4000.
>
>
> On Dec 29, 2016, at 19:48, haosdent <ha...@gmail.com> wrote:
>
> Hi, @Kiril It sounds like your executor reaches max open file limit. May
> you show the result of
>
> $ cat /proc/${your_executor_pid}/limits
>
>
>
> On Fri, Dec 30, 2016 at 1:07 AM, Kiril Menshikov <km...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I have executor which run java programs. One executor execute around 800
>> tasks. Last ~100 failed with "Too many open files". I increased nofile
>> and nproc limits. During my debug I could not say that problem in in the
>> tasks. But some times linux reach limits. I see some boxes are good with
>> out 'Too many open files errors'. But some has such errors.
>>
>> I run executor through mesos-containerizer and isolation is posix/cpu,
>> posix/mem.
>>
>> Can some one explain why this happens? Is it better to create separate
>> executor for each task? Tasks have common code but has different commands.
>>
>> Any help are welcomed.
>>
>> mesos-containerizer launch --command={"shell":true,"value":"java -cp
>> executor-all-1.0.jar com.stone.mesos.MesosTestExecutor"} --environment={"
>> LIBPROCESS_IP":"10.10.10.10","LIBPROCESS_PORT":"0","MESOS
>> _AGENT_ENDPOINT":"10.10.10.10:5051","MESOS_CHECKPOINT":"1","MESOS_DIRECTO
>> RY":"\/var\/lib\/mesos\/slaves\/7e30a916-1296-4f47-813
>> a-0972030b6907-S14\/frameworks\/7e30a916-1296-4f47-813a-
>> 0972030b6907-0020\/executors\/client_b12.tar-0f9e8f80-a217-
>> 4b28-bb5e-4dd7cc587381\/runs\/ee8857e2-ac19-4f07-810d-c2e71fbf522e","
>> MESOS_EXECUTOR_ID":"client_b12.tar-0f9e8f80-
>> a217-4b28-bb5e-4dd7cc587381","MESOS_EXECUTOR_SHUTDOWN_GRACE_
>> PERIOD":"5secs","MESOS_FRAMEWORK_ID":"7e30a916-1296-4f47-
>> 813a-0972030b6907-0020","MESOS_HTTP_COMMAND_EXECUTOR":"0","MESOS
>> _NATIVE_JAVA_LIBRARY":"\/usr\/local\/lib\/libmesos-1.1.0.so","MESOS
>> _NATIVE_LIBRARY":"\/usr\/local\/lib\/libmesos-1.1.0.so","MESOS_RECOVERY_
>> TIMEOUT":"15mins","MESOS_SANDBOX":"\/var\/lib\/mesos\/
>> slaves\/7e30a916-1296-4f47-813a-0972030b6907-S14\/framewo
>> rks\/7e30a916-1296-4f47-813a-0972030b6907-0020\/executors\/
>> client_b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381\/runs\/
>> ee8857e2-ac19-4f07-810d-c2e71fbf522e","MESOS_SLAVE_ID"
>> :"7e30a916-1296-4f47-813a-0972030b6907-S14","MESOS_SLAVE_PID":"slave(1)@
>> 10.10.10.10:5051","MESOS_SUBSCRIPTION_BACKOFF_MAX":"2secs","PATH":"\/
>> usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"}
>> --help=false --pipe_read=12 --pipe_write=13 --pre_exec_commands=[] --
>> runtime_directory=/var/run/mesos/containers/ee8857e2-ac19-4f07-810d-c2e71fbf522e
>> --unshare_namespace_mnt=false --user=ec2-user
>> --working_directory=/var/lib/mesos/slaves/7e30a916-1296-4f47
>> -813a-0972030b6907-S14/frameworks/7e30a916-1296-4f47-813a-
>> 0972030b6907-0020/executors/client_b12.tar-0f9e8f80-a217-
>> 4b28-bb5e-4dd7cc587381/runs/ee8857e2-ac19-4f07-810d-c2e71fbf522e
>>
>> Thanks,
>> -Kiril
>>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
>
>
--
Best Regards,
Haosdent Huang
Re: Too many files
Posted by Kiril Menshikov <km...@gmail.com>.
It’s 4000. Does this mean that sum of all child open files should not exceed this value? Because I see that task limit is also 4000.
> On Dec 29, 2016, at 19:48, haosdent <ha...@gmail.com> wrote:
>
> Hi, @Kiril It sounds like your executor reaches max open file limit. May you show the result of
>
> $ cat /proc/${your_executor_pid}/limits
>
>
>
> On Fri, Dec 30, 2016 at 1:07 AM, Kiril Menshikov <kmenshikov@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I have executor which run java programs. One executor execute around 800 tasks. Last ~100 failed with "Too many open files". I increased nofile and nproc limits. During my debug I could not say that problem in in the tasks. But some times linux reach limits. I see some boxes are good with out 'Too many open files errors'. But some has such errors.
>
> I run executor through mesos-containerizer and isolation is posix/cpu,posix/mem.
>
> Can some one explain why this happens? Is it better to create separate executor for each task? Tasks have common code but has different commands.
>
> Any help are welcomed.
>
> mesos-containerizer launch --command={"shell":true,"value":"java -cp executor-all-1.0.jar com.stone.mesos.MesosTestExecutor"} --environment={"LIBPROCESS_IP":"10.10.10.10","LIBPROCESS_PORT":"0","MESOS_AGENT_ENDPOINT":"10.10.10.10:5051 <http://10.10.10.10:5051/>","MESOS_CHECKPOINT":"1","MESOS_DIRECTORY":"\/var\/lib\/mesos\/slaves\/7e30a916-1296-4f47-813a-0972030b6907-S14\/frameworks\/7e30a916-1296-4f47-813a-0972030b6907-0020\/executors\/client_b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381\/runs\/ee8857e2-ac19-4f07-810d-c2e71fbf522e","MESOS_EXECUTOR_ID":"client_b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381","MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD":"5secs","MESOS_FRAMEWORK_ID":"7e30a916-1296-4f47-813a-0972030b6907-0020","MESOS_HTTP_COMMAND_EXECUTOR":"0","MESOS_NATIVE_JAVA_LIBRARY":"\/usr\/local\/lib\/libmesos-1.1.0.so <http://1.1.0.so/>","MESOS_NATIVE_LIBRARY":"\/usr\/local\/lib\/libmesos-1.1.0.so <http://1.1.0.so/>","MESOS_RECOVERY_TIMEOUT":"15mins","MESOS_SANDBOX":"\/var\/lib\/mesos\/slaves\/7e30a916-1296-4f47-813a-0972030b6907-S14\/frameworks\/7e30a916-1296-4f47-813a-0972030b6907-0020\/executors\/client_b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381\/runs\/ee8857e2-ac19-4f07-810d-c2e71fbf522e","MESOS_SLAVE_ID":"7e30a916-1296-4f47-813a-0972030b6907-S14","MESOS_SLAVE_PID":"slave(1)@10.10.10.10:5051 <http://10.10.10.10:5051/>","MESOS_SUBSCRIPTION_BACKOFF_MAX":"2secs","PATH":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"} --help=false --pipe_read=12 --pipe_write=13 --pre_exec_commands=[] --runtime_directory=/var/run/mesos/containers/ee8857e2-ac19-4f07-810d-c2e71fbf522e --unshare_namespace_mnt=false --user=ec2-user --working_directory=/var/lib/mesos/slaves/7e30a916-1296-4f47-813a-0972030b6907-S14/frameworks/7e30a916-1296-4f47-813a-0972030b6907-0020/executors/client_b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381/runs/ee8857e2-ac19-4f07-810d-c2e71fbf522e
>
> Thanks,
> -Kiril
>
>
>
> --
> Best Regards,
> Haosdent Huang
Re: Too many files
Posted by haosdent <ha...@gmail.com>.
Hi, @Kiril It sounds like your executor reaches max open file limit. May
you show the result of
$ cat /proc/${your_executor_pid}/limits
On Fri, Dec 30, 2016 at 1:07 AM, Kiril Menshikov <km...@gmail.com>
wrote:
> Hi,
>
> I have executor which run java programs. One executor execute around 800
> tasks. Last ~100 failed with "Too many open files". I increased nofile
> and nproc limits. During my debug I could not say that problem in in the
> tasks. But some times linux reach limits. I see some boxes are good with
> out 'Too many open files errors'. But some has such errors.
>
> I run executor through mesos-containerizer and isolation is posix/cpu,
> posix/mem.
>
> Can some one explain why this happens? Is it better to create separate
> executor for each task? Tasks have common code but has different commands.
>
> Any help are welcomed.
>
> mesos-containerizer launch --command={"shell":true,"value":"java -cp
> executor-all-1.0.jar com.stone.mesos.MesosTestExecutor"} --environment={"
> LIBPROCESS_IP":"10.10.10.10","LIBPROCESS_PORT":"0","MESOS_AGENT_
> ENDPOINT":"10.10.10.10:5051","MESOS_CHECKPOINT":"1","MESOS_
> DIRECTORY":"\/var\/lib\/mesos\/slaves\/7e30a916-1296-4f47-
> 813a-0972030b6907-S14\/frameworks\/7e30a916-1296-
> 4f47-813a-0972030b6907-0020\/executors\/client_b12.tar-
> 0f9e8f80-a217-4b28-bb5e-4dd7cc587381\/runs\/ee8857e2-
> ac19-4f07-810d-c2e71fbf522e","MESOS_EXECUTOR_ID":"client_
> b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381","MESOS_
> EXECUTOR_SHUTDOWN_GRACE_PERIOD":"5secs","MESOS_
> FRAMEWORK_ID":"7e30a916-1296-4f47-813a-0972030b6907-0020","MESOS
> _HTTP_COMMAND_EXECUTOR":"0","MESOS_NATIVE_JAVA_LIBRARY"
> :"\/usr\/local\/lib\/libmesos-1.1.0.so","MESOS_NATIVE_
> LIBRARY":"\/usr\/local\/lib\/libmesos-1.1.0.so","MESOS_
> RECOVERY_TIMEOUT":"15mins","MESOS_SANDBOX":"\/var\/lib\/
> mesos\/slaves\/7e30a916-1296-4f47-813a-0972030b6907-S14\/
> frameworks\/7e30a916-1296-4f47-813a-0972030b6907-0020\/
> executors\/client_b12.tar-0f9e8f80-a217-4b28-bb5e-
> 4dd7cc587381\/runs\/ee8857e2-ac19-4f07-810d-c2e71fbf522e","MESOS
> _SLAVE_ID":"7e30a916-1296-4f47-813a-0972030b6907-S14","MESOS_SLAVE_PID
> ":"slave(1)@10.10.10.10:5051","MESOS_SUBSCRIPTION_BACKOFF_MAX":"
> 2secs","PATH":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin
> :\/usr\/bin:\/sbin:\/bin"} --help=false --pipe_read=12 --pipe_write=13 --
> pre_exec_commands=[] --runtime_directory=/var/run/mesos
> /containers/ee8857e2-ac19-4f07-810d-c2e71fbf522e --unshare_namespace_mnt=false
> --user=ec2-user --working_directory=/var/lib/mesos/slaves/7e30a916-1296-
> 4f47-813a-0972030b6907-S14/frameworks/7e30a916-1296-4f47-
> 813a-0972030b6907-0020/executors/client_b12.tar-0f9e8f80-a217-4b28-bb5e-
> 4dd7cc587381/runs/ee8857e2-ac19-4f07-810d-c2e71fbf522e
>
> Thanks,
> -Kiril
>
--
Best Regards,
Haosdent Huang
Re: Too many files
Posted by Kiril Menshikov <km...@gmail.com>.
I found problems in systemd/linux.
First, I use wildcard in limit.conf which I changed to specific user.
Second, in service I define user and group to.
Because before even ulimit -n show correct value and process was launched
with correct user. Actual process limit was default.
Thanks,
-Kiril
On 30 December 2016 at 16:46, June Taylor <ju...@umn.edu> wrote:
> Kiril,
>
> We had this problem too and had a confusing time understanding how the
> ulimit setting was being applied to mesos executors.
>
> Here is what we learned which seems to work:
>
> The startup script /usr/bin/mesos-init-wrapper reads configuration values
> from /etc/default/mesos
>
> In this file, the configuration value for the ULIMIT parameter is fed as
> the argument to the 'ulimit' command.
>
> Therefore, this is what we have in our /etc/default/mesos file:
>
> ULIMIT="-n 20000"
>
> This gets passed in to the line in /usr/bin/mesos-init-wrapper which looks
> like this:
>
> [[ ! ${ULIMIT:-} ]] || ulimit $ULIMIT
>
> ..and sets your limits correctly. It is not sufficient to set systemwide
> ulimits.
>
> Do beware when performing upgrades that replace the content in
> /etc/default/mesos
>
>
> Thanks,
> June Taylor
> *(she / her)*
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Thu, Dec 29, 2016 at 11:07 AM, Kiril Menshikov <km...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I have executor which run java programs. One executor execute around 800
>> tasks. Last ~100 failed with "Too many open files". I increased nofile
>> and nproc limits. During my debug I could not say that problem in in the
>> tasks. But some times linux reach limits. I see some boxes are good with
>> out 'Too many open files errors'. But some has such errors.
>>
>> I run executor through mesos-containerizer and isolation is posix/cpu,
>> posix/mem.
>>
>> Can some one explain why this happens? Is it better to create separate
>> executor for each task? Tasks have common code but has different commands.
>>
>> Any help are welcomed.
>>
>> mesos-containerizer launch --command={"shell":true,"value":"java -cp
>> executor-all-1.0.jar com.stone.mesos.MesosTestExecutor"} --environment={"
>> LIBPROCESS_IP":"10.10.10.10","LIBPROCESS_PORT":"0","MESOS
>> _AGENT_ENDPOINT":"10.10.10.10:5051","MESOS_CHECKPOINT":"1","MESOS_DIRECTO
>> RY":"\/var\/lib\/mesos\/slaves\/7e30a916-1296-4f47-813
>> a-0972030b6907-S14\/frameworks\/7e30a916-1296-4f47-813a-
>> 0972030b6907-0020\/executors\/client_b12.tar-0f9e8f80-a217-
>> 4b28-bb5e-4dd7cc587381\/runs\/ee8857e2-ac19-4f07-810d-c2e71fbf522e","
>> MESOS_EXECUTOR_ID":"client_b12.tar-0f9e8f80-
>> a217-4b28-bb5e-4dd7cc587381","MESOS_EXECUTOR_SHUTDOWN_GRACE_
>> PERIOD":"5secs","MESOS_FRAMEWORK_ID":"7e30a916-1296-4f47-
>> 813a-0972030b6907-0020","MESOS_HTTP_COMMAND_EXECUTOR":"0","MESOS
>> _NATIVE_JAVA_LIBRARY":"\/usr\/local\/lib\/libmesos-1.1.0.so","MESOS
>> _NATIVE_LIBRARY":"\/usr\/local\/lib\/libmesos-1.1.0.so","MESOS_RECOVERY_
>> TIMEOUT":"15mins","MESOS_SANDBOX":"\/var\/lib\/mesos\/
>> slaves\/7e30a916-1296-4f47-813a-0972030b6907-S14\/framewo
>> rks\/7e30a916-1296-4f47-813a-0972030b6907-0020\/executors\/
>> client_b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381\/runs\/
>> ee8857e2-ac19-4f07-810d-c2e71fbf522e","MESOS_SLAVE_ID"
>> :"7e30a916-1296-4f47-813a-0972030b6907-S14","MESOS_SLAVE_PID":"slave(1)@
>> 10.10.10.10:5051","MESOS_SUBSCRIPTION_BACKOFF_MAX":"2secs","PATH":"\/
>> usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"}
>> --help=false --pipe_read=12 --pipe_write=13 --pre_exec_commands=[] --
>> runtime_directory=/var/run/mesos/containers/ee8857e2-ac19-4f07-810d-c2e71fbf522e
>> --unshare_namespace_mnt=false --user=ec2-user
>> --working_directory=/var/lib/mesos/slaves/7e30a916-1296-4f47
>> -813a-0972030b6907-S14/frameworks/7e30a916-1296-4f47-813a-
>> 0972030b6907-0020/executors/client_b12.tar-0f9e8f80-a217-
>> 4b28-bb5e-4dd7cc587381/runs/ee8857e2-ac19-4f07-810d-c2e71fbf522e
>>
>> Thanks,
>> -Kiril
>>
>
>
--
Thanks,
-Kiril
Phone +37126409291
Riga, Latvia
Skype perimetr122
Re: Too many files
Posted by June Taylor <ju...@umn.edu>.
Kiril,
We had this problem too and had a confusing time understanding how the
ulimit setting was being applied to mesos executors.
Here is what we learned which seems to work:
The startup script /usr/bin/mesos-init-wrapper reads configuration values
from /etc/default/mesos
In this file, the configuration value for the ULIMIT parameter is fed as
the argument to the 'ulimit' command.
Therefore, this is what we have in our /etc/default/mesos file:
ULIMIT="-n 20000"
This gets passed in to the line in /usr/bin/mesos-init-wrapper which looks
like this:
[[ ! ${ULIMIT:-} ]] || ulimit $ULIMIT
..and sets your limits correctly. It is not sufficient to set systemwide
ulimits.
Do beware when performing upgrades that replace the content in
/etc/default/mesos
Thanks,
June Taylor
*(she / her)*
System Administrator, Minnesota Population Center
University of Minnesota
On Thu, Dec 29, 2016 at 11:07 AM, Kiril Menshikov <km...@gmail.com>
wrote:
> Hi,
>
> I have executor which run java programs. One executor execute around 800
> tasks. Last ~100 failed with "Too many open files". I increased nofile
> and nproc limits. During my debug I could not say that problem in in the
> tasks. But some times linux reach limits. I see some boxes are good with
> out 'Too many open files errors'. But some has such errors.
>
> I run executor through mesos-containerizer and isolation is posix/cpu,
> posix/mem.
>
> Can some one explain why this happens? Is it better to create separate
> executor for each task? Tasks have common code but has different commands.
>
> Any help are welcomed.
>
> mesos-containerizer launch --command={"shell":true,"value":"java -cp
> executor-all-1.0.jar com.stone.mesos.MesosTestExecutor"} --environment={"
> LIBPROCESS_IP":"10.10.10.10","LIBPROCESS_PORT":"0","MESOS_AGENT_
> ENDPOINT":"10.10.10.10:5051","MESOS_CHECKPOINT":"1","MESOS_
> DIRECTORY":"\/var\/lib\/mesos\/slaves\/7e30a916-1296-4f47-
> 813a-0972030b6907-S14\/frameworks\/7e30a916-1296-
> 4f47-813a-0972030b6907-0020\/executors\/client_b12.tar-
> 0f9e8f80-a217-4b28-bb5e-4dd7cc587381\/runs\/ee8857e2-
> ac19-4f07-810d-c2e71fbf522e","MESOS_EXECUTOR_ID":"client_
> b12.tar-0f9e8f80-a217-4b28-bb5e-4dd7cc587381","MESOS_
> EXECUTOR_SHUTDOWN_GRACE_PERIOD":"5secs","MESOS_
> FRAMEWORK_ID":"7e30a916-1296-4f47-813a-0972030b6907-0020","MESOS
> _HTTP_COMMAND_EXECUTOR":"0","MESOS_NATIVE_JAVA_LIBRARY"
> :"\/usr\/local\/lib\/libmesos-1.1.0.so","MESOS_NATIVE_
> LIBRARY":"\/usr\/local\/lib\/libmesos-1.1.0.so","MESOS_
> RECOVERY_TIMEOUT":"15mins","MESOS_SANDBOX":"\/var\/lib\/
> mesos\/slaves\/7e30a916-1296-4f47-813a-0972030b6907-S14\/
> frameworks\/7e30a916-1296-4f47-813a-0972030b6907-0020\/
> executors\/client_b12.tar-0f9e8f80-a217-4b28-bb5e-
> 4dd7cc587381\/runs\/ee8857e2-ac19-4f07-810d-c2e71fbf522e","MESOS
> _SLAVE_ID":"7e30a916-1296-4f47-813a-0972030b6907-S14","MESOS_SLAVE_PID
> ":"slave(1)@10.10.10.10:5051","MESOS_SUBSCRIPTION_BACKOFF_MAX":"
> 2secs","PATH":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin
> :\/usr\/bin:\/sbin:\/bin"} --help=false --pipe_read=12 --pipe_write=13 --
> pre_exec_commands=[] --runtime_directory=/var/run/mesos
> /containers/ee8857e2-ac19-4f07-810d-c2e71fbf522e --unshare_namespace_mnt=false
> --user=ec2-user --working_directory=/var/lib/mesos/slaves/7e30a916-1296-
> 4f47-813a-0972030b6907-S14/frameworks/7e30a916-1296-4f47-
> 813a-0972030b6907-0020/executors/client_b12.tar-0f9e8f80-a217-4b28-bb5e-
> 4dd7cc587381/runs/ee8857e2-ac19-4f07-810d-c2e71fbf522e
>
> Thanks,
> -Kiril
>