You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Romi Kuntsman <ro...@totango.com> on 2014/11/06 08:15:11 UTC

Snappy temp files not cleaned up

Hello,

Spark has an internal cleanup mechanism
(defined by spark.cleaner.ttl, see
http://spark.apache.org/docs/latest/configuration.html)
which cleans up tasks and stages.

However, in our installation, we noticed that Snappy temporary files and
never cleaned up.

Is it a misconfiguration? Missing feature? How do you deal with build-up of
temp files?

Thanks,

*Romi Kuntsman*, *Big Data Engineer*
 http://www.totango.com

Re: Snappy temp files not cleaned up

Posted by Marius Soutier <mp...@gmail.com>.

Default value is infinite, so you need to enable it. Personally I’ve setup a couple of cron jobs to clean up /tmp and /var/run/spark.

On 06.11.2014, at 08:15, Romi Kuntsman <ro...@totango.com> wrote:

> Hello,
> 
> Spark has an internal cleanup mechanism
> (defined by spark.cleaner.ttl, see http://spark.apache.org/docs/latest/configuration.html)
> which cleans up tasks and stages.
> 
> However, in our installation, we noticed that Snappy temporary files and never cleaned up.
> 
> Is it a misconfiguration? Missing feature? How do you deal with build-up of temp files?
> 
> Thanks,
> 
> Romi Kuntsman, Big Data Engineer
> http://www.totango.com