You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Tom Arnfeld <to...@duedil.com> on 2014/09/06 17:23:02 UTC

Sandbox GC fails

I've noticed the disk on my mesos slaves filling up when running tasks that
generate large amounts of data in their sandbox directories (~2-5GB). The
tasks don't last very long, and I can see that the mesos GC process is
trying to delete them, but failing. Here are some logs;

------------------------------

W0906 02:56:00.256515  1434 gc.cpp:139] Failed to delete
'/var/lib/mesos-slave/slaves/20140724-124232-33820844-5050-18432-4/frameworks/20140724-201017-50598060-5050-16139-0314/executors/0c5ce2f8-3564-11e4-a014-22000a48a50a/runs/d3e1342f-89d8-4930-a9ed-35e33920779c':
Directory not empty
W0906 02:56:00.258396  1434 gc.cpp:139] Failed to delete
'/var/lib/mesos-slave/slaves/20140724-124232-33820844-5050-18432-4/frameworks/20140724-201017-50598060-5050-16139-0314':
Directory not empty
W0906 02:56:00.259904  1434 gc.cpp:139] Failed to delete
'/var/lib/mesos-slave/slaves/20140724-124232-33820844-5050-18432-4/frameworks/20140724-201017-50598060-5050-16139-0314/executors/0c5ce2f8-3564-11e4-a014-22000a48a50a':
Directory not empty

------------------------------

If I try and manually remove the directory mentioned, it works fine. Is
this a known issue, or should I do a little more debugging? I've not tried
to reproduce it under specific conditions yet.

As a side note, should mesos perhaps have some kind of retry mechanism for
GC? Also, will GC still run for an executor if the slave restarts after an
executor terminates but before the GC process runs?

Tom.

Re: Sandbox GC fails

Posted by Vinod Kone <vi...@gmail.com>.
os::rmdir() does recursive deletion. See it's implementation in stout. 

I would recommend patching that function to print more details (e.g., directory name and contents) to debug this. 

@vinodkone

> On Sep 11, 2014, at 8:32 AM, Tom Arnfeld <to...@duedil.com> wrote:
> 
> Hey Vinod,
> 
> So i've run into this a few more times and am struggling to understand why this is happening. It only seems to happen for some tasks.
> 
> From what I can tell, the path to the sandbox directory is scheduled for GC after the executor finishes. This GC process then iterates over the scheduled directories and figures out what needs to be cleaned up. Given a path that is to be removed, it then runs os::rmdir() on that directory. It doesn't seem to do anything explicitly recursive (maybe i'm looking in the wrong place) which is quite confusing, given the GC process is working fine for some tasks which have many levels of nested directors and files.
> 
> The log entry comes from this function; https://github.com/apache/mesos/blob/master/src/slave/gc.cpp#L127-L160 where I can see the os::rmdir() call.
> 
> Any chance you could bring some clarity to how recursive directory deletes happen? Assuming they do happen, the "Directory not empty" error is even more frustrating, because they're clearly not behaving correctly. Perhaps an error is being thrown when deleting the contents of the directory and that is being swallowed, so files still remain by the time the whole sandbox removal is attempted, causing a "Directory is not empty".
> 
> Appreciate any input!
> 
> 
>> On 8 September 2014 07:26, Tom Arnfeld <to...@duedil.com> wrote:
>> That's useful to know, thanks Vinod. I'll try and dig deeper.
>> 
>> 
>>> On Mon, Sep 8, 2014 at 5:33 AM, Vinod Kone <vi...@gmail.com> wrote:
>>> 
>>>> On Sat, Sep 6, 2014 at 8:23 AM, Tom Arnfeld <to...@duedil.com> wrote:
>>>> If I try and manually remove the directory mentioned, it works fine. Is this a known issue, or should I do a little more debugging? I've not tried to reproduce it under specific conditions yet.
>>>  
>>> This is surprising. GC does a recursive directory removal (see os::rmdir() in stout) using post-order traversal. Definitely some debugging is in order to see which directory failed and why. Does your sandbox contain any special files (other than directories and files) like mounts, devices etc?
>>> 
>>>  
>>>> As a side note, should mesos perhaps have some kind of retry mechanism for GC? Also, will GC still run for an executor if the slave restarts after an executor terminates but before the GC process runs?
>>> 
>>> I don't know what the error was above but I doubt a retry would've helped here. And yes GC runs for a terminated executor when slave restarts.
> 

Re: Sandbox GC fails

Posted by Tom Arnfeld <to...@duedil.com>.
Hey Vinod,

So i've run into this a few more times and am struggling to understand why
this is happening. It only seems to happen for some tasks.

>From what I can tell, the path to the sandbox directory is scheduled for GC
after the executor finishes. This GC process then iterates over the
scheduled directories and figures out what needs to be cleaned up. Given a
path that is to be removed, it then runs *os::rmdir()* on that directory.
It doesn't seem to do anything explicitly recursive (maybe i'm looking in
the wrong place) which is quite confusing, given the GC process is working
fine for some tasks which have many levels of nested directors and files.

The log entry comes from this function;
https://github.com/apache/mesos/blob/master/src/slave/gc.cpp#L127-L160
where I can see the *os::rmdir()* call.

Any chance you could bring some clarity to how recursive directory deletes
happen? Assuming they do happen, the "Directory not empty" error is even
more frustrating, because they're clearly not behaving correctly. Perhaps
an error is being thrown when deleting the contents of the directory and
that is being swallowed, so files still remain by the time the whole
sandbox removal is attempted, causing a "Directory is not empty".

Appreciate any input!


On 8 September 2014 07:26, Tom Arnfeld <to...@duedil.com> wrote:

> That's useful to know, thanks Vinod. I'll try and dig deeper.
>
>
> On Mon, Sep 8, 2014 at 5:33 AM, Vinod Kone <vi...@gmail.com> wrote:
>
>>
>> On Sat, Sep 6, 2014 at 8:23 AM, Tom Arnfeld <to...@duedil.com> wrote:
>>
>>> If I try and manually remove the directory mentioned, it works fine. Is
>>> this a known issue, or should I do a little more debugging? I've not tried
>>> to reproduce it under specific conditions yet.
>>>
>>>
>> This is surprising. GC does a recursive directory removal (see
>> os::rmdir() in stout) using post-order traversal. Definitely some debugging
>> is in order to see which directory failed and why. Does your sandbox
>> contain any special files (other than directories and files) like mounts,
>> devices etc?
>>
>>
>>
>>>  As a side note, should mesos perhaps have some kind of retry mechanism
>>> for GC? Also, will GC still run for an executor if the slave restarts after
>>> an executor terminates but before the GC process runs?
>>>
>>
>> I don't know what the error was above but I doubt a retry would've helped
>> here. And yes GC runs for a terminated executor when slave restarts.
>>
>
>

Re: Sandbox GC fails

Posted by Tom Arnfeld <to...@duedil.com>.
That's useful to know, thanks Vinod. I'll try and dig deeper.

On Mon, Sep 8, 2014 at 5:33 AM, Vinod Kone <vi...@gmail.com> wrote:

> On Sat, Sep 6, 2014 at 8:23 AM, Tom Arnfeld <to...@duedil.com> wrote:
>> If I try and manually remove the directory mentioned, it works fine. Is
>> this a known issue, or should I do a little more debugging? I've not tried
>> to reproduce it under specific conditions yet.
>>
>>
> This is surprising. GC does a recursive directory removal (see os::rmdir()
> in stout) using post-order traversal. Definitely some debugging is in order
> to see which directory failed and why. Does your sandbox contain any
> special files (other than directories and files) like mounts, devices etc?
>> As a side note, should mesos perhaps have some kind of retry mechanism for
>> GC? Also, will GC still run for an executor if the slave restarts after an
>> executor terminates but before the GC process runs?
>>
> I don't know what the error was above but I doubt a retry would've helped
> here. And yes GC runs for a terminated executor when slave restarts.

Re: Sandbox GC fails

Posted by Vinod Kone <vi...@gmail.com>.
On Sat, Sep 6, 2014 at 8:23 AM, Tom Arnfeld <to...@duedil.com> wrote:

> If I try and manually remove the directory mentioned, it works fine. Is
> this a known issue, or should I do a little more debugging? I've not tried
> to reproduce it under specific conditions yet.
>
>
This is surprising. GC does a recursive directory removal (see os::rmdir()
in stout) using post-order traversal. Definitely some debugging is in order
to see which directory failed and why. Does your sandbox contain any
special files (other than directories and files) like mounts, devices etc?



> As a side note, should mesos perhaps have some kind of retry mechanism for
> GC? Also, will GC still run for an executor if the slave restarts after an
> executor terminates but before the GC process runs?
>

I don't know what the error was above but I doubt a retry would've helped
here. And yes GC runs for a terminated executor when slave restarts.