You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Michele Bertasi <mi...@brightcomputing.com> on 2017/03/22 14:55:28 UTC
Detect when mesos agent needs work directory cleanup
Hi everybody,
I'm having troubles with the cleanup of the mesos agent work directory.
When we change some configuration parameters to the agent (e.g. the
--resource flag), the agent refuses to start and asks the removal of
*/var/lib/mesos-agent/meta/slaves/latest* in the logs.
Since we are automating these changes, we would like to know when to
perform this cleanup programmatically.
Right now we are doing that *after every single change*, but that's not
ideal, since every time mesos master assigns a new ID to the slave. It
seems unnecessary to me.
My questions are:
- is there a way to know when this cleanup is needed (without looking at
the logs)?
- do we have problems if we clean up the work directory every time the
node boots (so the configuration might or might not have been changed at
all)?
Thanks, kind regards,
Mike
Re: Detect when mesos agent needs work directory cleanup
Posted by Michele Bertasi <mi...@brightcomputing.com>.
I tested the command and it works. Thanks!
On Fri, Mar 24, 2017 at 2:24 AM, Benjamin Mahler <bm...@apache.org> wrote:
> I would recommend avoiding a manual clean up of the work directory, since
> it's not guaranteed that this approach will remain correct as things
> evolve. To have the agent perform the cleanup using its own logic, you can
> run:
>
> mesos-agent --recover=cleanup --work_dir=<dir> --master=<master>
>
> Also, there is already reboot handling in place to ensure the agent
> doesn't bother recovering if the node has rebooted since it last ran,
> although I believe this may be in a future release. +yan, neil
>
> On Thu, Mar 23, 2017 at 1:11 AM, Michele Bertasi <michele.bertasi@
> brightcomputing.com> wrote:
>
>> Thank you. I will do that then.
>>
>> Mike
>>
>> On Wed, Mar 22, 2017 at 6:57 PM, Tomek Janiszewski <ja...@gmail.com>
>> wrote:
>>
>>> 1. Cleanup in required when agent configuration is not compatible with
>>> previous version. This mean task runtime change. This occurs when:
>>> resources, isolators, attributes or containerizers change.
>>> 2. IMO it's OK. When you reboot node all its tasks are gone so you won't
>>> lose anything. Node will be registered as a new node.
>>>
>>> śr., 22.03.2017, 15:55 użytkownik Michele Bertasi <
>>> michele.bertasi@brightcomputing.com> napisał:
>>>
>>>> Hi everybody,
>>>>
>>>> I'm having troubles with the cleanup of the mesos agent work directory.
>>>> When we change some configuration parameters to the agent (e.g. the
>>>> --resource flag), the agent refuses to start and asks the removal of
>>>> */var/lib/mesos-agent/meta/slaves/latest* in the logs.
>>>>
>>>> Since we are automating these changes, we would like to know when to
>>>> perform this cleanup programmatically.
>>>>
>>>> Right now we are doing that *after every single change*, but that's
>>>> not ideal, since every time mesos master assigns a new ID to the slave. It
>>>> seems unnecessary to me.
>>>>
>>>> My questions are:
>>>>
>>>> - is there a way to know when this cleanup is needed (without
>>>> looking at the logs)?
>>>> - do we have problems if we clean up the work directory every time
>>>> the node boots (so the configuration might or might not have been changed
>>>> at all)?
>>>>
>>>>
>>>> Thanks, kind regards,
>>>>
>>>> Mike
>>>>
>>>
>>
>
Re: Detect when mesos agent needs work directory cleanup
Posted by Benjamin Mahler <bm...@apache.org>.
I would recommend avoiding a manual clean up of the work directory, since
it's not guaranteed that this approach will remain correct as things
evolve. To have the agent perform the cleanup using its own logic, you can
run:
mesos-agent --recover=cleanup --work_dir=<dir> --master=<master>
Also, there is already reboot handling in place to ensure the agent doesn't
bother recovering if the node has rebooted since it last ran, although I
believe this may be in a future release. +yan, neil
On Thu, Mar 23, 2017 at 1:11 AM, Michele Bertasi <
michele.bertasi@brightcomputing.com> wrote:
> Thank you. I will do that then.
>
> Mike
>
> On Wed, Mar 22, 2017 at 6:57 PM, Tomek Janiszewski <ja...@gmail.com>
> wrote:
>
>> 1. Cleanup in required when agent configuration is not compatible with
>> previous version. This mean task runtime change. This occurs when:
>> resources, isolators, attributes or containerizers change.
>> 2. IMO it's OK. When you reboot node all its tasks are gone so you won't
>> lose anything. Node will be registered as a new node.
>>
>> śr., 22.03.2017, 15:55 użytkownik Michele Bertasi <
>> michele.bertasi@brightcomputing.com> napisał:
>>
>>> Hi everybody,
>>>
>>> I'm having troubles with the cleanup of the mesos agent work directory.
>>> When we change some configuration parameters to the agent (e.g. the
>>> --resource flag), the agent refuses to start and asks the removal of
>>> */var/lib/mesos-agent/meta/slaves/latest* in the logs.
>>>
>>> Since we are automating these changes, we would like to know when to
>>> perform this cleanup programmatically.
>>>
>>> Right now we are doing that *after every single change*, but that's not
>>> ideal, since every time mesos master assigns a new ID to the slave. It
>>> seems unnecessary to me.
>>>
>>> My questions are:
>>>
>>> - is there a way to know when this cleanup is needed (without
>>> looking at the logs)?
>>> - do we have problems if we clean up the work directory every time
>>> the node boots (so the configuration might or might not have been changed
>>> at all)?
>>>
>>>
>>> Thanks, kind regards,
>>>
>>> Mike
>>>
>>
>
Re: Detect when mesos agent needs work directory cleanup
Posted by Michele Bertasi <mi...@brightcomputing.com>.
Thank you. I will do that then.
Mike
On Wed, Mar 22, 2017 at 6:57 PM, Tomek Janiszewski <ja...@gmail.com>
wrote:
> 1. Cleanup in required when agent configuration is not compatible with
> previous version. This mean task runtime change. This occurs when:
> resources, isolators, attributes or containerizers change.
> 2. IMO it's OK. When you reboot node all its tasks are gone so you won't
> lose anything. Node will be registered as a new node.
>
> śr., 22.03.2017, 15:55 użytkownik Michele Bertasi <michele.bertasi@
> brightcomputing.com> napisał:
>
>> Hi everybody,
>>
>> I'm having troubles with the cleanup of the mesos agent work directory.
>> When we change some configuration parameters to the agent (e.g. the
>> --resource flag), the agent refuses to start and asks the removal of
>> */var/lib/mesos-agent/meta/slaves/latest* in the logs.
>>
>> Since we are automating these changes, we would like to know when to
>> perform this cleanup programmatically.
>>
>> Right now we are doing that *after every single change*, but that's not
>> ideal, since every time mesos master assigns a new ID to the slave. It
>> seems unnecessary to me.
>>
>> My questions are:
>>
>> - is there a way to know when this cleanup is needed (without looking
>> at the logs)?
>> - do we have problems if we clean up the work directory every time
>> the node boots (so the configuration might or might not have been changed
>> at all)?
>>
>>
>> Thanks, kind regards,
>>
>> Mike
>>
>
Re: Detect when mesos agent needs work directory cleanup
Posted by Tomek Janiszewski <ja...@gmail.com>.
1. Cleanup in required when agent configuration is not compatible with
previous version. This mean task runtime change. This occurs when:
resources, isolators, attributes or containerizers change.
2. IMO it's OK. When you reboot node all its tasks are gone so you won't
lose anything. Node will be registered as a new node.
śr., 22.03.2017, 15:55 użytkownik Michele Bertasi <
michele.bertasi@brightcomputing.com> napisał:
> Hi everybody,
>
> I'm having troubles with the cleanup of the mesos agent work directory.
> When we change some configuration parameters to the agent (e.g. the
> --resource flag), the agent refuses to start and asks the removal of
> */var/lib/mesos-agent/meta/slaves/latest* in the logs.
>
> Since we are automating these changes, we would like to know when to
> perform this cleanup programmatically.
>
> Right now we are doing that *after every single change*, but that's not
> ideal, since every time mesos master assigns a new ID to the slave. It
> seems unnecessary to me.
>
> My questions are:
>
> - is there a way to know when this cleanup is needed (without looking
> at the logs)?
> - do we have problems if we clean up the work directory every time the
> node boots (so the configuration might or might not have been changed at
> all)?
>
>
> Thanks, kind regards,
>
> Mike
>