You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shuporno Choudhury <sh...@gmail.com> on 2018/08/01 07:19:47 UTC
Clearing usercache on EMR [pyspark]
Hi everyone,
I am running spark jobs on EMR (using pyspark). I noticed that after
running jobs, the size of the usercache (basically the filecache folder)
keeps on increasing (with directory names as 1,2,3,4,5,...).
Directory location: */mnt/yarn/usercache/hadoop/**filecache/*
Is there a way to avoid creating these directories or automatically
clearing the usercache/filecache after a job/periodically?
--
--Thanks,
Shuporno Choudhury
Re: Clearing usercache on EMR [pyspark]
Posted by Shuporno Choudhury <sh...@gmail.com>.
Can anyone please help me with this issue?
On Fri, 3 Aug 2018 at 11:27, Shuporno Choudhury <
shuporno.choudhury@gmail.com> wrote:
> Can anyone please help me with this issue?
>
> On Wed, 1 Aug 2018 at 12:50, Shuporno Choudhury [via Apache Spark User
> List] <ml...@n3.nabble.com> wrote:
>
>> Hi everyone,
>> I am running spark jobs on EMR (using pyspark). I noticed that after
>> running jobs, the size of the usercache (basically the filecache folder)
>> keeps on increasing (with directory names as 1,2,3,4,5,...).
>> Directory location: */mnt/yarn/usercache/hadoop/**filecache/*
>> Is there a way to avoid creating these directories or automatically
>> clearing the usercache/filecache after a job/periodically?
>> --
>> --Thanks,
>> Shuporno Choudhury
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/Clearing-usercache-on-EMR-pyspark-tp33096.html
>> To start a new topic under Apache Spark User List, email
>> ml+s1001560n1h17@n3.nabble.com
>> To unsubscribe from Apache Spark User List, click here
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=c2h1cG9ybm8uY2hvdWRodXJ5QGdtYWlsLmNvbXwxfC0xODI0MTU0MzQ0>
>> .
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> --
> --Thanks,
> Shuporno Choudhury
>
--
--Thanks,
Shuporno Choudhury