You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by June Taylor <ju...@umn.edu> on 2016/04/25 18:42:33 UTC
Setting ulimits on mesos-slave
We are operating a 6-node cluster running on Ubuntu, and have noticed that
the ulimit settings within the slave context are difficult to set and
predict.
The documentation is a bit unclear on this point, as well.
We have had some luck adding a configuration line to
/etc/init/mesos-slave.conf as follows:
limit nofile 20000 20000
limit fsize unlimited unlimited
The nofile limit seems to be respected, however the fsize limit does not.
It is also mysterious that the system-wide limits are not inherited by the
slave process. We would prefer to set all of these system-wide and have
mesos-slave observe them.
Can you please advise where you are setting your ulimits for the
mesos-slave if it is working for you?
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
Re: Setting ulimits on mesos-slave
Posted by June Taylor <ju...@umn.edu>.
Here is an update and a solution for this thread:
The /usr/bin/mesos-init-wrapper script looks for configuration settings in
a number of different places. This isn't necessarily a problem, but it did
make finding a solution difficult, since people have different approaches
for where they provide configuration settings in their deployments. In
addition, the ulimit parameter is not handled quite like the other
configuration parameters, because the value of the parameter is literally
passed as the argument to the `ulimit` command. What works is:
Within /etc/default/mesos file, have a line in this format:
ULIMIT="-n 20000"
...where the content within the double-quotation marks is exactly what you
would, as the root user, supply as the argument string to simply running
`ulimit` directly.
This also complicates upgrades because the content of /etc/default/* may be
overwritten during an upgrade. It would be preferable to specify the ulimit
configuration within /etc/mesos/ which would not be overwritten. However,
unlike the other parameters, this doesn't seem to be sourced by the
/usr/bin/mesos-init-wrapper script.
So, we have a solution, and perhaps this is helpful for others and is
useful to the consideration of package maintainers.
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Tue, Apr 26, 2016 at 11:51 AM, June Taylor <ju...@umn.edu> wrote:
> That is a good point - is the executor spun off as a different PID, if so
> it would perhaps inherit a different fsize value.
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Tue, Apr 26, 2016 at 11:43 AM, haosdent <ha...@gmail.com> wrote:
>
>> I disable the limit globally and retry set
>>
>> ```
>> limit fsize unlimited unlimited
>> ```
>>
>> in /etc/init/mesos-slave.conf after reboot. It works as well according to
>> check limits under /proc May you double check this in your side? I wonder
>> if the fszie only effective on Mesos Agent and doesn't work in your Mesos
>> executor.
>>
>> On Tue, Apr 26, 2016 at 8:55 PM, June Taylor <ju...@umn.edu> wrote:
>>
>>> Perhaps the "unlimited" keyword is what is not working? What should be
>>> specified for Unlimited?
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Mon, Apr 25, 2016 at 9:48 PM, haosdent <ha...@gmail.com> wrote:
>>>
>>>> According my test, it works in my side.
>>>>
>>>> * Before add it to /etc/init/mesos-slave.conf
>>>>
>>>> ```
>>>> cat /proc/16550/limits
>>>> Limit Soft Limit Hard Limit
>>>> Units
>>>> Max cpu time unlimited unlimited
>>>> seconds
>>>> Max file size unlimited unlimited
>>>> bytes
>>>> Max data size unlimited unlimited
>>>> bytes
>>>> Max stack size 10485760 unlimited
>>>> bytes
>>>> Max core file size 0 unlimited
>>>> bytes
>>>> Max resident set unlimited unlimited
>>>> bytes
>>>> Max processes 63655 63655
>>>> processes
>>>> Max open files 8192 8192
>>>> files
>>>> Max locked memory 65536 65536
>>>> bytes
>>>> Max address space unlimited unlimited
>>>> bytes
>>>> Max file locks unlimited unlimited
>>>> locks
>>>> Max pending signals 63655 63655
>>>> signals
>>>> Max msgqueue size 819200 819200
>>>> bytes
>>>> Max nice priority 0 0
>>>> Max realtime priority 0 0
>>>> Max realtime timeout unlimited unlimited us
>>>> ```
>>>>
>>>> * After add it to /etc/init/mesos-slave.conf
>>>>
>>>> ```
>>>> description "mesos slave"
>>>>
>>>> # I didn't use ulimit because I have already set ulimit globally.
>>>> limit fsize 20001 20001
>>>> limit nofile 20000 20000
>>>> ```
>>>>
>>>> ```
>>>> cat /proc/16602/limits
>>>> Limit Soft Limit Hard Limit
>>>> Units
>>>> Max cpu time unlimited unlimited
>>>> seconds
>>>> Max file size 20001 20001
>>>> bytes # <- Have changed
>>>> Max data size unlimited unlimited
>>>> bytes
>>>> Max stack size 10485760 unlimited
>>>> bytes
>>>> Max core file size 0 unlimited
>>>> bytes
>>>> Max resident set unlimited unlimited
>>>> bytes
>>>> Max processes 63655 63655
>>>> processes
>>>> Max open files 8192 8192
>>>> files
>>>> Max locked memory 65536 65536
>>>> bytes
>>>> Max address space unlimited unlimited
>>>> bytes
>>>> Max file locks unlimited unlimited
>>>> locks
>>>> Max pending signals 63655 63655
>>>> signals
>>>> Max msgqueue size 819200 819200
>>>> bytes
>>>> Max nice priority 0 0
>>>> Max realtime priority 0 0
>>>> Max realtime timeout unlimited unlimited us
>>>> ```
>>>>
>>>> On Tue, Apr 26, 2016 at 3:51 AM, June Taylor <ju...@umn.edu> wrote:
>>>>
>>>>> Hello. We are running it as root, and it is able to specify the ulimit
>>>>> for open files, as noted in our config file, but it is not setting the File
>>>>> Size limit, which is remaining at 8MB.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> June Taylor
>>>>> System Administrator, Minnesota Population Center
>>>>> University of Minnesota
>>>>>
>>>>> On Mon, Apr 25, 2016 at 2:50 PM, Dick Davies <di...@hellooperator.net>
>>>>> wrote:
>>>>>
>>>>>> Hi June
>>>>>>
>>>>>> are you running Mesos as root, or a non-privileged user? Non-root
>>>>>> won't be able to up their own ulimit too high
>>>>>> (sorry, not an upstart expert as RHELs is laughably incomplete).
>>>>>>
>>>>>> On 25 April 2016 at 19:15, June Taylor <ju...@umn.edu> wrote:
>>>>>> > What I'm saying is even putting them within the upstart script, per
>>>>>> the
>>>>>> > Mesos documentation, isn't working for the file block limit. We're
>>>>>> still
>>>>>> > getting 8MB useable, and as a result executors fail when attempting
>>>>>> to write
>>>>>> > larger files.
>>>>>> >
>>>>>> >
>>>>>> > Thanks,
>>>>>> > June Taylor
>>>>>> > System Administrator, Minnesota Population Center
>>>>>> > University of Minnesota
>>>>>> >
>>>>>> > On Mon, Apr 25, 2016 at 11:53 AM, haosdent <ha...@gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> If you set in your upstart script, it isn't system wide and only
>>>>>> effective
>>>>>> >> in that session. I think need change /etc/security/limits.conf and
>>>>>> >> /etc/sysctl.conf to make your ulimit work globally.
>>>>>> >>
>>>>>> >> On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <ju...@umn.edu>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Somewhere an 8MB maximum file size is being applied on just one
>>>>>> of our
>>>>>> >>> slaves, for example.
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> Thanks,
>>>>>> >>> June Taylor
>>>>>> >>> System Administrator, Minnesota Population Center
>>>>>> >>> University of Minnesota
>>>>>> >>>
>>>>>> >>> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> We are operating a 6-node cluster running on Ubuntu, and have
>>>>>> noticed
>>>>>> >>>> that the ulimit settings within the slave context are difficult
>>>>>> to set and
>>>>>> >>>> predict.
>>>>>> >>>>
>>>>>> >>>> The documentation is a bit unclear on this point, as well.
>>>>>> >>>>
>>>>>> >>>> We have had some luck adding a configuration line to
>>>>>> >>>> /etc/init/mesos-slave.conf as follows:
>>>>>> >>>> limit nofile 20000 20000
>>>>>> >>>> limit fsize unlimited unlimited
>>>>>> >>>>
>>>>>> >>>> The nofile limit seems to be respected, however the fsize limit
>>>>>> does
>>>>>> >>>> not.
>>>>>> >>>>
>>>>>> >>>> It is also mysterious that the system-wide limits are not
>>>>>> inherited by
>>>>>> >>>> the slave process. We would prefer to set all of these
>>>>>> system-wide and have
>>>>>> >>>> mesos-slave observe them.
>>>>>> >>>>
>>>>>> >>>> Can you please advise where you are setting your ulimits for the
>>>>>> >>>> mesos-slave if it is working for you?
>>>>>> >>>>
>>>>>> >>>> Thanks,
>>>>>> >>>> June Taylor
>>>>>> >>>> System Administrator, Minnesota Population Center
>>>>>> >>>> University of Minnesota
>>>>>> >>>
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Best Regards,
>>>>>> >> Haosdent Huang
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Haosdent Huang
>>>>
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>
Re: Setting ulimits on mesos-slave
Posted by June Taylor <ju...@umn.edu>.
That is a good point - is the executor spun off as a different PID, if so
it would perhaps inherit a different fsize value.
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Tue, Apr 26, 2016 at 11:43 AM, haosdent <ha...@gmail.com> wrote:
> I disable the limit globally and retry set
>
> ```
> limit fsize unlimited unlimited
> ```
>
> in /etc/init/mesos-slave.conf after reboot. It works as well according to
> check limits under /proc May you double check this in your side? I wonder
> if the fszie only effective on Mesos Agent and doesn't work in your Mesos
> executor.
>
> On Tue, Apr 26, 2016 at 8:55 PM, June Taylor <ju...@umn.edu> wrote:
>
>> Perhaps the "unlimited" keyword is what is not working? What should be
>> specified for Unlimited?
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Mon, Apr 25, 2016 at 9:48 PM, haosdent <ha...@gmail.com> wrote:
>>
>>> According my test, it works in my side.
>>>
>>> * Before add it to /etc/init/mesos-slave.conf
>>>
>>> ```
>>> cat /proc/16550/limits
>>> Limit Soft Limit Hard Limit Units
>>> Max cpu time unlimited unlimited
>>> seconds
>>> Max file size unlimited unlimited bytes
>>> Max data size unlimited unlimited bytes
>>> Max stack size 10485760 unlimited bytes
>>> Max core file size 0 unlimited bytes
>>> Max resident set unlimited unlimited bytes
>>> Max processes 63655 63655
>>> processes
>>> Max open files 8192 8192 files
>>> Max locked memory 65536 65536 bytes
>>> Max address space unlimited unlimited bytes
>>> Max file locks unlimited unlimited locks
>>> Max pending signals 63655 63655
>>> signals
>>> Max msgqueue size 819200 819200 bytes
>>> Max nice priority 0 0
>>> Max realtime priority 0 0
>>> Max realtime timeout unlimited unlimited us
>>> ```
>>>
>>> * After add it to /etc/init/mesos-slave.conf
>>>
>>> ```
>>> description "mesos slave"
>>>
>>> # I didn't use ulimit because I have already set ulimit globally.
>>> limit fsize 20001 20001
>>> limit nofile 20000 20000
>>> ```
>>>
>>> ```
>>> cat /proc/16602/limits
>>> Limit Soft Limit Hard Limit Units
>>> Max cpu time unlimited unlimited
>>> seconds
>>> Max file size 20001 20001
>>> bytes # <- Have changed
>>> Max data size unlimited unlimited bytes
>>> Max stack size 10485760 unlimited bytes
>>> Max core file size 0 unlimited bytes
>>> Max resident set unlimited unlimited bytes
>>> Max processes 63655 63655
>>> processes
>>> Max open files 8192 8192 files
>>> Max locked memory 65536 65536 bytes
>>> Max address space unlimited unlimited bytes
>>> Max file locks unlimited unlimited locks
>>> Max pending signals 63655 63655
>>> signals
>>> Max msgqueue size 819200 819200 bytes
>>> Max nice priority 0 0
>>> Max realtime priority 0 0
>>> Max realtime timeout unlimited unlimited us
>>> ```
>>>
>>> On Tue, Apr 26, 2016 at 3:51 AM, June Taylor <ju...@umn.edu> wrote:
>>>
>>>> Hello. We are running it as root, and it is able to specify the ulimit
>>>> for open files, as noted in our config file, but it is not setting the File
>>>> Size limit, which is remaining at 8MB.
>>>>
>>>>
>>>> Thanks,
>>>> June Taylor
>>>> System Administrator, Minnesota Population Center
>>>> University of Minnesota
>>>>
>>>> On Mon, Apr 25, 2016 at 2:50 PM, Dick Davies <di...@hellooperator.net>
>>>> wrote:
>>>>
>>>>> Hi June
>>>>>
>>>>> are you running Mesos as root, or a non-privileged user? Non-root
>>>>> won't be able to up their own ulimit too high
>>>>> (sorry, not an upstart expert as RHELs is laughably incomplete).
>>>>>
>>>>> On 25 April 2016 at 19:15, June Taylor <ju...@umn.edu> wrote:
>>>>> > What I'm saying is even putting them within the upstart script, per
>>>>> the
>>>>> > Mesos documentation, isn't working for the file block limit. We're
>>>>> still
>>>>> > getting 8MB useable, and as a result executors fail when attempting
>>>>> to write
>>>>> > larger files.
>>>>> >
>>>>> >
>>>>> > Thanks,
>>>>> > June Taylor
>>>>> > System Administrator, Minnesota Population Center
>>>>> > University of Minnesota
>>>>> >
>>>>> > On Mon, Apr 25, 2016 at 11:53 AM, haosdent <ha...@gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> If you set in your upstart script, it isn't system wide and only
>>>>> effective
>>>>> >> in that session. I think need change /etc/security/limits.conf and
>>>>> >> /etc/sysctl.conf to make your ulimit work globally.
>>>>> >>
>>>>> >> On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <ju...@umn.edu> wrote:
>>>>> >>>
>>>>> >>> Somewhere an 8MB maximum file size is being applied on just one of
>>>>> our
>>>>> >>> slaves, for example.
>>>>> >>>
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>> June Taylor
>>>>> >>> System Administrator, Minnesota Population Center
>>>>> >>> University of Minnesota
>>>>> >>>
>>>>> >>> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>> We are operating a 6-node cluster running on Ubuntu, and have
>>>>> noticed
>>>>> >>>> that the ulimit settings within the slave context are difficult
>>>>> to set and
>>>>> >>>> predict.
>>>>> >>>>
>>>>> >>>> The documentation is a bit unclear on this point, as well.
>>>>> >>>>
>>>>> >>>> We have had some luck adding a configuration line to
>>>>> >>>> /etc/init/mesos-slave.conf as follows:
>>>>> >>>> limit nofile 20000 20000
>>>>> >>>> limit fsize unlimited unlimited
>>>>> >>>>
>>>>> >>>> The nofile limit seems to be respected, however the fsize limit
>>>>> does
>>>>> >>>> not.
>>>>> >>>>
>>>>> >>>> It is also mysterious that the system-wide limits are not
>>>>> inherited by
>>>>> >>>> the slave process. We would prefer to set all of these
>>>>> system-wide and have
>>>>> >>>> mesos-slave observe them.
>>>>> >>>>
>>>>> >>>> Can you please advise where you are setting your ulimits for the
>>>>> >>>> mesos-slave if it is working for you?
>>>>> >>>>
>>>>> >>>> Thanks,
>>>>> >>>> June Taylor
>>>>> >>>> System Administrator, Minnesota Population Center
>>>>> >>>> University of Minnesota
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Haosdent Huang
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Setting ulimits on mesos-slave
Posted by haosdent <ha...@gmail.com>.
I disable the limit globally and retry set
```
limit fsize unlimited unlimited
```
in /etc/init/mesos-slave.conf after reboot. It works as well according to
check limits under /proc May you double check this in your side? I wonder
if the fszie only effective on Mesos Agent and doesn't work in your Mesos
executor.
On Tue, Apr 26, 2016 at 8:55 PM, June Taylor <ju...@umn.edu> wrote:
> Perhaps the "unlimited" keyword is what is not working? What should be
> specified for Unlimited?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Mon, Apr 25, 2016 at 9:48 PM, haosdent <ha...@gmail.com> wrote:
>
>> According my test, it works in my side.
>>
>> * Before add it to /etc/init/mesos-slave.conf
>>
>> ```
>> cat /proc/16550/limits
>> Limit Soft Limit Hard Limit Units
>> Max cpu time unlimited unlimited
>> seconds
>> Max file size unlimited unlimited bytes
>> Max data size unlimited unlimited bytes
>> Max stack size 10485760 unlimited bytes
>> Max core file size 0 unlimited bytes
>> Max resident set unlimited unlimited bytes
>> Max processes 63655 63655
>> processes
>> Max open files 8192 8192 files
>> Max locked memory 65536 65536 bytes
>> Max address space unlimited unlimited bytes
>> Max file locks unlimited unlimited locks
>> Max pending signals 63655 63655
>> signals
>> Max msgqueue size 819200 819200 bytes
>> Max nice priority 0 0
>> Max realtime priority 0 0
>> Max realtime timeout unlimited unlimited us
>> ```
>>
>> * After add it to /etc/init/mesos-slave.conf
>>
>> ```
>> description "mesos slave"
>>
>> # I didn't use ulimit because I have already set ulimit globally.
>> limit fsize 20001 20001
>> limit nofile 20000 20000
>> ```
>>
>> ```
>> cat /proc/16602/limits
>> Limit Soft Limit Hard Limit Units
>> Max cpu time unlimited unlimited
>> seconds
>> Max file size 20001 20001 bytes
>> # <- Have changed
>> Max data size unlimited unlimited bytes
>> Max stack size 10485760 unlimited bytes
>> Max core file size 0 unlimited bytes
>> Max resident set unlimited unlimited bytes
>> Max processes 63655 63655
>> processes
>> Max open files 8192 8192 files
>> Max locked memory 65536 65536 bytes
>> Max address space unlimited unlimited bytes
>> Max file locks unlimited unlimited locks
>> Max pending signals 63655 63655
>> signals
>> Max msgqueue size 819200 819200 bytes
>> Max nice priority 0 0
>> Max realtime priority 0 0
>> Max realtime timeout unlimited unlimited us
>> ```
>>
>> On Tue, Apr 26, 2016 at 3:51 AM, June Taylor <ju...@umn.edu> wrote:
>>
>>> Hello. We are running it as root, and it is able to specify the ulimit
>>> for open files, as noted in our config file, but it is not setting the File
>>> Size limit, which is remaining at 8MB.
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Mon, Apr 25, 2016 at 2:50 PM, Dick Davies <di...@hellooperator.net>
>>> wrote:
>>>
>>>> Hi June
>>>>
>>>> are you running Mesos as root, or a non-privileged user? Non-root
>>>> won't be able to up their own ulimit too high
>>>> (sorry, not an upstart expert as RHELs is laughably incomplete).
>>>>
>>>> On 25 April 2016 at 19:15, June Taylor <ju...@umn.edu> wrote:
>>>> > What I'm saying is even putting them within the upstart script, per
>>>> the
>>>> > Mesos documentation, isn't working for the file block limit. We're
>>>> still
>>>> > getting 8MB useable, and as a result executors fail when attempting
>>>> to write
>>>> > larger files.
>>>> >
>>>> >
>>>> > Thanks,
>>>> > June Taylor
>>>> > System Administrator, Minnesota Population Center
>>>> > University of Minnesota
>>>> >
>>>> > On Mon, Apr 25, 2016 at 11:53 AM, haosdent <ha...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> If you set in your upstart script, it isn't system wide and only
>>>> effective
>>>> >> in that session. I think need change /etc/security/limits.conf and
>>>> >> /etc/sysctl.conf to make your ulimit work globally.
>>>> >>
>>>> >> On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <ju...@umn.edu> wrote:
>>>> >>>
>>>> >>> Somewhere an 8MB maximum file size is being applied on just one of
>>>> our
>>>> >>> slaves, for example.
>>>> >>>
>>>> >>>
>>>> >>> Thanks,
>>>> >>> June Taylor
>>>> >>> System Administrator, Minnesota Population Center
>>>> >>> University of Minnesota
>>>> >>>
>>>> >>> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu> wrote:
>>>> >>>>
>>>> >>>> We are operating a 6-node cluster running on Ubuntu, and have
>>>> noticed
>>>> >>>> that the ulimit settings within the slave context are difficult to
>>>> set and
>>>> >>>> predict.
>>>> >>>>
>>>> >>>> The documentation is a bit unclear on this point, as well.
>>>> >>>>
>>>> >>>> We have had some luck adding a configuration line to
>>>> >>>> /etc/init/mesos-slave.conf as follows:
>>>> >>>> limit nofile 20000 20000
>>>> >>>> limit fsize unlimited unlimited
>>>> >>>>
>>>> >>>> The nofile limit seems to be respected, however the fsize limit
>>>> does
>>>> >>>> not.
>>>> >>>>
>>>> >>>> It is also mysterious that the system-wide limits are not
>>>> inherited by
>>>> >>>> the slave process. We would prefer to set all of these system-wide
>>>> and have
>>>> >>>> mesos-slave observe them.
>>>> >>>>
>>>> >>>> Can you please advise where you are setting your ulimits for the
>>>> >>>> mesos-slave if it is working for you?
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>> June Taylor
>>>> >>>> System Administrator, Minnesota Population Center
>>>> >>>> University of Minnesota
>>>> >>>
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Haosdent Huang
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>
--
Best Regards,
Haosdent Huang
Re: Setting ulimits on mesos-slave
Posted by June Taylor <ju...@umn.edu>.
Perhaps the "unlimited" keyword is what is not working? What should be
specified for Unlimited?
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Mon, Apr 25, 2016 at 9:48 PM, haosdent <ha...@gmail.com> wrote:
> According my test, it works in my side.
>
> * Before add it to /etc/init/mesos-slave.conf
>
> ```
> cat /proc/16550/limits
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size unlimited unlimited bytes
> Max data size unlimited unlimited bytes
> Max stack size 10485760 unlimited bytes
> Max core file size 0 unlimited bytes
> Max resident set unlimited unlimited bytes
> Max processes 63655 63655
> processes
> Max open files 8192 8192 files
> Max locked memory 65536 65536 bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 63655 63655 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
> ```
>
> * After add it to /etc/init/mesos-slave.conf
>
> ```
> description "mesos slave"
>
> # I didn't use ulimit because I have already set ulimit globally.
> limit fsize 20001 20001
> limit nofile 20000 20000
> ```
>
> ```
> cat /proc/16602/limits
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size 20001 20001 bytes
> # <- Have changed
> Max data size unlimited unlimited bytes
> Max stack size 10485760 unlimited bytes
> Max core file size 0 unlimited bytes
> Max resident set unlimited unlimited bytes
> Max processes 63655 63655
> processes
> Max open files 8192 8192 files
> Max locked memory 65536 65536 bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 63655 63655 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
> ```
>
> On Tue, Apr 26, 2016 at 3:51 AM, June Taylor <ju...@umn.edu> wrote:
>
>> Hello. We are running it as root, and it is able to specify the ulimit
>> for open files, as noted in our config file, but it is not setting the File
>> Size limit, which is remaining at 8MB.
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Mon, Apr 25, 2016 at 2:50 PM, Dick Davies <di...@hellooperator.net>
>> wrote:
>>
>>> Hi June
>>>
>>> are you running Mesos as root, or a non-privileged user? Non-root
>>> won't be able to up their own ulimit too high
>>> (sorry, not an upstart expert as RHELs is laughably incomplete).
>>>
>>> On 25 April 2016 at 19:15, June Taylor <ju...@umn.edu> wrote:
>>> > What I'm saying is even putting them within the upstart script, per the
>>> > Mesos documentation, isn't working for the file block limit. We're
>>> still
>>> > getting 8MB useable, and as a result executors fail when attempting to
>>> write
>>> > larger files.
>>> >
>>> >
>>> > Thanks,
>>> > June Taylor
>>> > System Administrator, Minnesota Population Center
>>> > University of Minnesota
>>> >
>>> > On Mon, Apr 25, 2016 at 11:53 AM, haosdent <ha...@gmail.com> wrote:
>>> >>
>>> >> If you set in your upstart script, it isn't system wide and only
>>> effective
>>> >> in that session. I think need change /etc/security/limits.conf and
>>> >> /etc/sysctl.conf to make your ulimit work globally.
>>> >>
>>> >> On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <ju...@umn.edu> wrote:
>>> >>>
>>> >>> Somewhere an 8MB maximum file size is being applied on just one of
>>> our
>>> >>> slaves, for example.
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>> June Taylor
>>> >>> System Administrator, Minnesota Population Center
>>> >>> University of Minnesota
>>> >>>
>>> >>> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu> wrote:
>>> >>>>
>>> >>>> We are operating a 6-node cluster running on Ubuntu, and have
>>> noticed
>>> >>>> that the ulimit settings within the slave context are difficult to
>>> set and
>>> >>>> predict.
>>> >>>>
>>> >>>> The documentation is a bit unclear on this point, as well.
>>> >>>>
>>> >>>> We have had some luck adding a configuration line to
>>> >>>> /etc/init/mesos-slave.conf as follows:
>>> >>>> limit nofile 20000 20000
>>> >>>> limit fsize unlimited unlimited
>>> >>>>
>>> >>>> The nofile limit seems to be respected, however the fsize limit does
>>> >>>> not.
>>> >>>>
>>> >>>> It is also mysterious that the system-wide limits are not inherited
>>> by
>>> >>>> the slave process. We would prefer to set all of these system-wide
>>> and have
>>> >>>> mesos-slave observe them.
>>> >>>>
>>> >>>> Can you please advise where you are setting your ulimits for the
>>> >>>> mesos-slave if it is working for you?
>>> >>>>
>>> >>>> Thanks,
>>> >>>> June Taylor
>>> >>>> System Administrator, Minnesota Population Center
>>> >>>> University of Minnesota
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards,
>>> >> Haosdent Huang
>>> >
>>> >
>>>
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Setting ulimits on mesos-slave
Posted by haosdent <ha...@gmail.com>.
According my test, it works in my side.
* Before add it to /etc/init/mesos-slave.conf
```
cat /proc/16550/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 63655 63655
processes
Max open files 8192 8192 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 63655 63655 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
```
* After add it to /etc/init/mesos-slave.conf
```
description "mesos slave"
# I didn't use ulimit because I have already set ulimit globally.
limit fsize 20001 20001
limit nofile 20000 20000
```
```
cat /proc/16602/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size 20001 20001 bytes
# <- Have changed
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 63655 63655
processes
Max open files 8192 8192 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 63655 63655 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
```
On Tue, Apr 26, 2016 at 3:51 AM, June Taylor <ju...@umn.edu> wrote:
> Hello. We are running it as root, and it is able to specify the ulimit for
> open files, as noted in our config file, but it is not setting the File
> Size limit, which is remaining at 8MB.
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Mon, Apr 25, 2016 at 2:50 PM, Dick Davies <di...@hellooperator.net>
> wrote:
>
>> Hi June
>>
>> are you running Mesos as root, or a non-privileged user? Non-root
>> won't be able to up their own ulimit too high
>> (sorry, not an upstart expert as RHELs is laughably incomplete).
>>
>> On 25 April 2016 at 19:15, June Taylor <ju...@umn.edu> wrote:
>> > What I'm saying is even putting them within the upstart script, per the
>> > Mesos documentation, isn't working for the file block limit. We're still
>> > getting 8MB useable, and as a result executors fail when attempting to
>> write
>> > larger files.
>> >
>> >
>> > Thanks,
>> > June Taylor
>> > System Administrator, Minnesota Population Center
>> > University of Minnesota
>> >
>> > On Mon, Apr 25, 2016 at 11:53 AM, haosdent <ha...@gmail.com> wrote:
>> >>
>> >> If you set in your upstart script, it isn't system wide and only
>> effective
>> >> in that session. I think need change /etc/security/limits.conf and
>> >> /etc/sysctl.conf to make your ulimit work globally.
>> >>
>> >> On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <ju...@umn.edu> wrote:
>> >>>
>> >>> Somewhere an 8MB maximum file size is being applied on just one of our
>> >>> slaves, for example.
>> >>>
>> >>>
>> >>> Thanks,
>> >>> June Taylor
>> >>> System Administrator, Minnesota Population Center
>> >>> University of Minnesota
>> >>>
>> >>> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu> wrote:
>> >>>>
>> >>>> We are operating a 6-node cluster running on Ubuntu, and have noticed
>> >>>> that the ulimit settings within the slave context are difficult to
>> set and
>> >>>> predict.
>> >>>>
>> >>>> The documentation is a bit unclear on this point, as well.
>> >>>>
>> >>>> We have had some luck adding a configuration line to
>> >>>> /etc/init/mesos-slave.conf as follows:
>> >>>> limit nofile 20000 20000
>> >>>> limit fsize unlimited unlimited
>> >>>>
>> >>>> The nofile limit seems to be respected, however the fsize limit does
>> >>>> not.
>> >>>>
>> >>>> It is also mysterious that the system-wide limits are not inherited
>> by
>> >>>> the slave process. We would prefer to set all of these system-wide
>> and have
>> >>>> mesos-slave observe them.
>> >>>>
>> >>>> Can you please advise where you are setting your ulimits for the
>> >>>> mesos-slave if it is working for you?
>> >>>>
>> >>>> Thanks,
>> >>>> June Taylor
>> >>>> System Administrator, Minnesota Population Center
>> >>>> University of Minnesota
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards,
>> >> Haosdent Huang
>> >
>> >
>>
>
>
--
Best Regards,
Haosdent Huang
Re: Setting ulimits on mesos-slave
Posted by June Taylor <ju...@umn.edu>.
Hello. We are running it as root, and it is able to specify the ulimit for
open files, as noted in our config file, but it is not setting the File
Size limit, which is remaining at 8MB.
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Mon, Apr 25, 2016 at 2:50 PM, Dick Davies <di...@hellooperator.net> wrote:
> Hi June
>
> are you running Mesos as root, or a non-privileged user? Non-root
> won't be able to up their own ulimit too high
> (sorry, not an upstart expert as RHELs is laughably incomplete).
>
> On 25 April 2016 at 19:15, June Taylor <ju...@umn.edu> wrote:
> > What I'm saying is even putting them within the upstart script, per the
> > Mesos documentation, isn't working for the file block limit. We're still
> > getting 8MB useable, and as a result executors fail when attempting to
> write
> > larger files.
> >
> >
> > Thanks,
> > June Taylor
> > System Administrator, Minnesota Population Center
> > University of Minnesota
> >
> > On Mon, Apr 25, 2016 at 11:53 AM, haosdent <ha...@gmail.com> wrote:
> >>
> >> If you set in your upstart script, it isn't system wide and only
> effective
> >> in that session. I think need change /etc/security/limits.conf and
> >> /etc/sysctl.conf to make your ulimit work globally.
> >>
> >> On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <ju...@umn.edu> wrote:
> >>>
> >>> Somewhere an 8MB maximum file size is being applied on just one of our
> >>> slaves, for example.
> >>>
> >>>
> >>> Thanks,
> >>> June Taylor
> >>> System Administrator, Minnesota Population Center
> >>> University of Minnesota
> >>>
> >>> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu> wrote:
> >>>>
> >>>> We are operating a 6-node cluster running on Ubuntu, and have noticed
> >>>> that the ulimit settings within the slave context are difficult to
> set and
> >>>> predict.
> >>>>
> >>>> The documentation is a bit unclear on this point, as well.
> >>>>
> >>>> We have had some luck adding a configuration line to
> >>>> /etc/init/mesos-slave.conf as follows:
> >>>> limit nofile 20000 20000
> >>>> limit fsize unlimited unlimited
> >>>>
> >>>> The nofile limit seems to be respected, however the fsize limit does
> >>>> not.
> >>>>
> >>>> It is also mysterious that the system-wide limits are not inherited by
> >>>> the slave process. We would prefer to set all of these system-wide
> and have
> >>>> mesos-slave observe them.
> >>>>
> >>>> Can you please advise where you are setting your ulimits for the
> >>>> mesos-slave if it is working for you?
> >>>>
> >>>> Thanks,
> >>>> June Taylor
> >>>> System Administrator, Minnesota Population Center
> >>>> University of Minnesota
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Haosdent Huang
> >
> >
>
Re: Setting ulimits on mesos-slave
Posted by Dick Davies <di...@hellooperator.net>.
Hi June
are you running Mesos as root, or a non-privileged user? Non-root
won't be able to up their own ulimit too high
(sorry, not an upstart expert as RHELs is laughably incomplete).
On 25 April 2016 at 19:15, June Taylor <ju...@umn.edu> wrote:
> What I'm saying is even putting them within the upstart script, per the
> Mesos documentation, isn't working for the file block limit. We're still
> getting 8MB useable, and as a result executors fail when attempting to write
> larger files.
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Mon, Apr 25, 2016 at 11:53 AM, haosdent <ha...@gmail.com> wrote:
>>
>> If you set in your upstart script, it isn't system wide and only effective
>> in that session. I think need change /etc/security/limits.conf and
>> /etc/sysctl.conf to make your ulimit work globally.
>>
>> On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <ju...@umn.edu> wrote:
>>>
>>> Somewhere an 8MB maximum file size is being applied on just one of our
>>> slaves, for example.
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu> wrote:
>>>>
>>>> We are operating a 6-node cluster running on Ubuntu, and have noticed
>>>> that the ulimit settings within the slave context are difficult to set and
>>>> predict.
>>>>
>>>> The documentation is a bit unclear on this point, as well.
>>>>
>>>> We have had some luck adding a configuration line to
>>>> /etc/init/mesos-slave.conf as follows:
>>>> limit nofile 20000 20000
>>>> limit fsize unlimited unlimited
>>>>
>>>> The nofile limit seems to be respected, however the fsize limit does
>>>> not.
>>>>
>>>> It is also mysterious that the system-wide limits are not inherited by
>>>> the slave process. We would prefer to set all of these system-wide and have
>>>> mesos-slave observe them.
>>>>
>>>> Can you please advise where you are setting your ulimits for the
>>>> mesos-slave if it is working for you?
>>>>
>>>> Thanks,
>>>> June Taylor
>>>> System Administrator, Minnesota Population Center
>>>> University of Minnesota
>>>
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>
>
Re: Setting ulimits on mesos-slave
Posted by June Taylor <ju...@umn.edu>.
What I'm saying is even putting them within the upstart script, per the
Mesos documentation, isn't working for the file block limit. We're still
getting 8MB useable, and as a result executors fail when attempting to
write larger files.
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Mon, Apr 25, 2016 at 11:53 AM, haosdent <ha...@gmail.com> wrote:
> If you set in your upstart script, it isn't system wide and only effective
> in that session. I think need change /etc/security/limits.conf
> and /etc/sysctl.conf to make your ulimit work globally.
>
> On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <ju...@umn.edu> wrote:
>
>> Somewhere an 8MB maximum file size is being applied on just one of our
>> slaves, for example.
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu> wrote:
>>
>>> We are operating a 6-node cluster running on Ubuntu, and have noticed
>>> that the ulimit settings within the slave context are difficult to set and
>>> predict.
>>>
>>> The documentation is a bit unclear on this point, as well.
>>>
>>> We have had some luck adding a configuration line to
>>> /etc/init/mesos-slave.conf as follows:
>>> limit nofile 20000 20000
>>> limit fsize unlimited unlimited
>>>
>>> The nofile limit seems to be respected, however the fsize limit does not.
>>>
>>> It is also mysterious that the system-wide limits are not inherited by
>>> the slave process. We would prefer to set all of these system-wide and have
>>> mesos-slave observe them.
>>>
>>> Can you please advise where you are setting your ulimits for the
>>> mesos-slave if it is working for you?
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Setting ulimits on mesos-slave
Posted by haosdent <ha...@gmail.com>.
If you set in your upstart script, it isn't system wide and only effective
in that session. I think need change /etc/security/limits.conf
and /etc/sysctl.conf to make your ulimit work globally.
On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <ju...@umn.edu> wrote:
> Somewhere an 8MB maximum file size is being applied on just one of our
> slaves, for example.
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu> wrote:
>
>> We are operating a 6-node cluster running on Ubuntu, and have noticed
>> that the ulimit settings within the slave context are difficult to set and
>> predict.
>>
>> The documentation is a bit unclear on this point, as well.
>>
>> We have had some luck adding a configuration line to
>> /etc/init/mesos-slave.conf as follows:
>> limit nofile 20000 20000
>> limit fsize unlimited unlimited
>>
>> The nofile limit seems to be respected, however the fsize limit does not.
>>
>> It is also mysterious that the system-wide limits are not inherited by
>> the slave process. We would prefer to set all of these system-wide and have
>> mesos-slave observe them.
>>
>> Can you please advise where you are setting your ulimits for the
>> mesos-slave if it is working for you?
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>
>
--
Best Regards,
Haosdent Huang
Re: Setting ulimits on mesos-slave
Posted by June Taylor <ju...@umn.edu>.
Somewhere an 8MB maximum file size is being applied on just one of our
slaves, for example.
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu> wrote:
> We are operating a 6-node cluster running on Ubuntu, and have noticed that
> the ulimit settings within the slave context are difficult to set and
> predict.
>
> The documentation is a bit unclear on this point, as well.
>
> We have had some luck adding a configuration line to
> /etc/init/mesos-slave.conf as follows:
> limit nofile 20000 20000
> limit fsize unlimited unlimited
>
> The nofile limit seems to be respected, however the fsize limit does not.
>
> It is also mysterious that the system-wide limits are not inherited by the
> slave process. We would prefer to set all of these system-wide and have
> mesos-slave observe them.
>
> Can you please advise where you are setting your ulimits for the
> mesos-slave if it is working for you?
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>