You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by June Taylor <ju...@umn.edu> on 2016/05/25 13:04:51 UTC
Re: Setting ulimits on mesos-slave
Here is an update and a solution for this thread:
The /usr/bin/mesos-init-wrapper script looks for configuration settings in
a number of different places. This isn't necessarily a problem, but it did
make finding a solution difficult, since people have different approaches
for where they provide configuration settings in their deployments. In
addition, the ulimit parameter is not handled quite like the other
configuration parameters, because the value of the parameter is literally
passed as the argument to the `ulimit` command. What works is:
Within /etc/default/mesos file, have a line in this format:
ULIMIT="-n 20000"
...where the content within the double-quotation marks is exactly what you
would, as the root user, supply as the argument string to simply running
`ulimit` directly.
This also complicates upgrades because the content of /etc/default/* may be
overwritten during an upgrade. It would be preferable to specify the ulimit
configuration within /etc/mesos/ which would not be overwritten. However,
unlike the other parameters, this doesn't seem to be sourced by the
/usr/bin/mesos-init-wrapper script.
So, we have a solution, and perhaps this is helpful for others and is
useful to the consideration of package maintainers.
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Tue, Apr 26, 2016 at 11:51 AM, June Taylor <ju...@umn.edu> wrote:
> That is a good point - is the executor spun off as a different PID, if so
> it would perhaps inherit a different fsize value.
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Tue, Apr 26, 2016 at 11:43 AM, haosdent <ha...@gmail.com> wrote:
>
>> I disable the limit globally and retry set
>>
>> ```
>> limit fsize unlimited unlimited
>> ```
>>
>> in /etc/init/mesos-slave.conf after reboot. It works as well according to
>> check limits under /proc May you double check this in your side? I wonder
>> if the fszie only effective on Mesos Agent and doesn't work in your Mesos
>> executor.
>>
>> On Tue, Apr 26, 2016 at 8:55 PM, June Taylor <ju...@umn.edu> wrote:
>>
>>> Perhaps the "unlimited" keyword is what is not working? What should be
>>> specified for Unlimited?
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Mon, Apr 25, 2016 at 9:48 PM, haosdent <ha...@gmail.com> wrote:
>>>
>>>> According my test, it works in my side.
>>>>
>>>> * Before add it to /etc/init/mesos-slave.conf
>>>>
>>>> ```
>>>> cat /proc/16550/limits
>>>> Limit Soft Limit Hard Limit
>>>> Units
>>>> Max cpu time unlimited unlimited
>>>> seconds
>>>> Max file size unlimited unlimited
>>>> bytes
>>>> Max data size unlimited unlimited
>>>> bytes
>>>> Max stack size 10485760 unlimited
>>>> bytes
>>>> Max core file size 0 unlimited
>>>> bytes
>>>> Max resident set unlimited unlimited
>>>> bytes
>>>> Max processes 63655 63655
>>>> processes
>>>> Max open files 8192 8192
>>>> files
>>>> Max locked memory 65536 65536
>>>> bytes
>>>> Max address space unlimited unlimited
>>>> bytes
>>>> Max file locks unlimited unlimited
>>>> locks
>>>> Max pending signals 63655 63655
>>>> signals
>>>> Max msgqueue size 819200 819200
>>>> bytes
>>>> Max nice priority 0 0
>>>> Max realtime priority 0 0
>>>> Max realtime timeout unlimited unlimited us
>>>> ```
>>>>
>>>> * After add it to /etc/init/mesos-slave.conf
>>>>
>>>> ```
>>>> description "mesos slave"
>>>>
>>>> # I didn't use ulimit because I have already set ulimit globally.
>>>> limit fsize 20001 20001
>>>> limit nofile 20000 20000
>>>> ```
>>>>
>>>> ```
>>>> cat /proc/16602/limits
>>>> Limit Soft Limit Hard Limit
>>>> Units
>>>> Max cpu time unlimited unlimited
>>>> seconds
>>>> Max file size 20001 20001
>>>> bytes # <- Have changed
>>>> Max data size unlimited unlimited
>>>> bytes
>>>> Max stack size 10485760 unlimited
>>>> bytes
>>>> Max core file size 0 unlimited
>>>> bytes
>>>> Max resident set unlimited unlimited
>>>> bytes
>>>> Max processes 63655 63655
>>>> processes
>>>> Max open files 8192 8192
>>>> files
>>>> Max locked memory 65536 65536
>>>> bytes
>>>> Max address space unlimited unlimited
>>>> bytes
>>>> Max file locks unlimited unlimited
>>>> locks
>>>> Max pending signals 63655 63655
>>>> signals
>>>> Max msgqueue size 819200 819200
>>>> bytes
>>>> Max nice priority 0 0
>>>> Max realtime priority 0 0
>>>> Max realtime timeout unlimited unlimited us
>>>> ```
>>>>
>>>> On Tue, Apr 26, 2016 at 3:51 AM, June Taylor <ju...@umn.edu> wrote:
>>>>
>>>>> Hello. We are running it as root, and it is able to specify the ulimit
>>>>> for open files, as noted in our config file, but it is not setting the File
>>>>> Size limit, which is remaining at 8MB.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> June Taylor
>>>>> System Administrator, Minnesota Population Center
>>>>> University of Minnesota
>>>>>
>>>>> On Mon, Apr 25, 2016 at 2:50 PM, Dick Davies <di...@hellooperator.net>
>>>>> wrote:
>>>>>
>>>>>> Hi June
>>>>>>
>>>>>> are you running Mesos as root, or a non-privileged user? Non-root
>>>>>> won't be able to up their own ulimit too high
>>>>>> (sorry, not an upstart expert as RHELs is laughably incomplete).
>>>>>>
>>>>>> On 25 April 2016 at 19:15, June Taylor <ju...@umn.edu> wrote:
>>>>>> > What I'm saying is even putting them within the upstart script, per
>>>>>> the
>>>>>> > Mesos documentation, isn't working for the file block limit. We're
>>>>>> still
>>>>>> > getting 8MB useable, and as a result executors fail when attempting
>>>>>> to write
>>>>>> > larger files.
>>>>>> >
>>>>>> >
>>>>>> > Thanks,
>>>>>> > June Taylor
>>>>>> > System Administrator, Minnesota Population Center
>>>>>> > University of Minnesota
>>>>>> >
>>>>>> > On Mon, Apr 25, 2016 at 11:53 AM, haosdent <ha...@gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> If you set in your upstart script, it isn't system wide and only
>>>>>> effective
>>>>>> >> in that session. I think need change /etc/security/limits.conf and
>>>>>> >> /etc/sysctl.conf to make your ulimit work globally.
>>>>>> >>
>>>>>> >> On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <ju...@umn.edu>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Somewhere an 8MB maximum file size is being applied on just one
>>>>>> of our
>>>>>> >>> slaves, for example.
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> Thanks,
>>>>>> >>> June Taylor
>>>>>> >>> System Administrator, Minnesota Population Center
>>>>>> >>> University of Minnesota
>>>>>> >>>
>>>>>> >>> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <ju...@umn.edu>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> We are operating a 6-node cluster running on Ubuntu, and have
>>>>>> noticed
>>>>>> >>>> that the ulimit settings within the slave context are difficult
>>>>>> to set and
>>>>>> >>>> predict.
>>>>>> >>>>
>>>>>> >>>> The documentation is a bit unclear on this point, as well.
>>>>>> >>>>
>>>>>> >>>> We have had some luck adding a configuration line to
>>>>>> >>>> /etc/init/mesos-slave.conf as follows:
>>>>>> >>>> limit nofile 20000 20000
>>>>>> >>>> limit fsize unlimited unlimited
>>>>>> >>>>
>>>>>> >>>> The nofile limit seems to be respected, however the fsize limit
>>>>>> does
>>>>>> >>>> not.
>>>>>> >>>>
>>>>>> >>>> It is also mysterious that the system-wide limits are not
>>>>>> inherited by
>>>>>> >>>> the slave process. We would prefer to set all of these
>>>>>> system-wide and have
>>>>>> >>>> mesos-slave observe them.
>>>>>> >>>>
>>>>>> >>>> Can you please advise where you are setting your ulimits for the
>>>>>> >>>> mesos-slave if it is working for you?
>>>>>> >>>>
>>>>>> >>>> Thanks,
>>>>>> >>>> June Taylor
>>>>>> >>>> System Administrator, Minnesota Population Center
>>>>>> >>>> University of Minnesota
>>>>>> >>>
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Best Regards,
>>>>>> >> Haosdent Huang
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Haosdent Huang
>>>>
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>