You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Mingyu Kim <mk...@palantir.com> on 2014/02/06 00:46:44 UTC

Clean up app metadata on worker nodes

After creating a lot of Spark connections, work/app-* folders in Worker
nodes keep getting created without any clean-up being done. This
particularly becomes a problem when the Spark driver programs ship jars or
files. Is there any way to garbage collect these without manually deleting
them? Thanks!

Mingyu

Re: Clean up app metadata on worker nodes

Posted by Andrew Ash <an...@andrewash.com>.

I'm observing this as well on 0.9.0, with several 10s of GB accumulating in
that directory but never being cleaned up.  I think this has gotten more
pronounced in 0.9.0 as well with large reducers spilling to disk.

On Wed, Feb 5, 2014 at 3:46 PM, Mingyu Kim <mk...@palantir.com> wrote:

> After creating a lot of Spark connections, work/app-* folders in Worker
> nodes keep getting created without any clean-up being done. This
> particularly becomes a problem when the Spark driver programs ship jars or
> files. Is there any way to garbage collect these without manually deleting
> them? Thanks!
>
> Mingyu
>