You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ayan guha <gu...@gmail.com> on 2016/07/13 03:54:57 UTC
Inode for STS
Hi
We are running Spark Thrift Server as a long running application. However,
it looks like it is filling up /tmp/hive folder with lots of small files
and directories with no file in them, blowing out inode limit and
preventing any connection with "No Space Left in Device" issue.
What is the best way to clean up those small files periodically?
--
Best Regards,
Ayan Guha
Re: Inode for STS
Posted by ayan guha <gu...@gmail.com>.
Hi
Thanks for this. However, I am interested in regular deletion of temp while
server is up. Additionally, the link says it is not of use for multi-user
environment. Any other idea? is there any variation of cleaner.ttl?
On Mon, Jul 18, 2016 at 8:00 PM, Chanh Le <gi...@gmail.com> wrote:
> Hi Ayan,
> I seem like you mention this
> https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.start.cleanup.scratchdir
> Default it was set false by default.
>
>
>
>
>
> On Jul 13, 2016, at 5:01 PM, ayan guha <gu...@gmail.com> wrote:
>
> Thanks Christophe. Any comment from Spark dev community member would
> really helpful on the Jira.
>
> What I saw today is shutting down the thrift server process lead to a
> clean up. Also, we started removing any empty folders from /tmp. Is there
> any other or better method?
>
> On Wed, Jul 13, 2016 at 5:25 PM, Christophe Préaud <
> christophe.preaud@kelkoo.com> wrote:
>
>> Hi Ayan,
>>
>> I have opened a JIRA about this issues, but there are no answer so far:
>> SPARK-15401 <https://issues.apache.org/jira/browse/SPARK-15401>
>>
>> Regards,
>> Christophe.
>>
>>
>> On 13/07/16 05:54, ayan guha wrote:
>>
>> Hi
>>
>> We are running Spark Thrift Server as a long running application.
>> However, it looks like it is filling up /tmp/hive folder with lots of
>> small files and directories with no file in them, blowing out inode limit
>> and preventing any connection with "No Space Left in Device" issue.
>>
>> What is the best way to clean up those small files periodically?
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>>
>>
>> ------------------------------
>> Kelkoo SAS
>> Société par Actions Simplifiée
>> Au capital de € 4.168.964,30
>> Siège social : 158 Ter Rue du Temple 75003 Paris
>> 425 093 069 RCS Paris
>>
>> Ce message et les pièces jointes sont confidentiels et établis à
>> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
>> destinataire de ce message, merci de le détruire et d'en avertir
>> l'expéditeur.
>>
>
>
>
> --
> Best Regards,
> Ayan Guha
>
>
>
--
Best Regards,
Ayan Guha
Re: Inode for STS
Posted by Chanh Le <gi...@gmail.com>.
Hi Ayan,
I seem like you mention this https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.start.cleanup.scratchdir <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.start.cleanup.scratchdir>
Default it was set false by default.
> On Jul 13, 2016, at 5:01 PM, ayan guha <gu...@gmail.com> wrote:
>
> Thanks Christophe. Any comment from Spark dev community member would really helpful on the Jira.
>
> What I saw today is shutting down the thrift server process lead to a clean up. Also, we started removing any empty folders from /tmp. Is there any other or better method?
>
> On Wed, Jul 13, 2016 at 5:25 PM, Christophe Préaud <christophe.preaud@kelkoo.com <ma...@kelkoo.com>> wrote:
> Hi Ayan,
>
> I have opened a JIRA about this issues, but there are no answer so far: SPARK-15401 <https://issues.apache.org/jira/browse/SPARK-15401>
>
> Regards,
> Christophe.
>
>
> On 13/07/16 05:54, ayan guha wrote:
>> Hi
>>
>> We are running Spark Thrift Server as a long running application. However, it looks like it is filling up /tmp/hive folder with lots of small files and directories with no file in them, blowing out inode limit and preventing any connection with "No Space Left in Device" issue.
>>
>> What is the best way to clean up those small files periodically?
>>
>> --
>> Best Regards,
>> Ayan Guha
>
>
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 158 Ter Rue du Temple 75003 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.
>
>
>
> --
> Best Regards,
> Ayan Guha
Re: Inode for STS
Posted by ayan guha <gu...@gmail.com>.
Thanks Christophe. Any comment from Spark dev community member would really
helpful on the Jira.
What I saw today is shutting down the thrift server process lead to a clean
up. Also, we started removing any empty folders from /tmp. Is there any
other or better method?
On Wed, Jul 13, 2016 at 5:25 PM, Christophe Préaud <
christophe.preaud@kelkoo.com> wrote:
> Hi Ayan,
>
> I have opened a JIRA about this issues, but there are no answer so far:
> SPARK-15401 <https://issues.apache.org/jira/browse/SPARK-15401>
>
> Regards,
> Christophe.
>
>
> On 13/07/16 05:54, ayan guha wrote:
>
> Hi
>
> We are running Spark Thrift Server as a long running application. However,
> it looks like it is filling up /tmp/hive folder with lots of small files
> and directories with no file in them, blowing out inode limit and
> preventing any connection with "No Space Left in Device" issue.
>
> What is the best way to clean up those small files periodically?
>
> --
> Best Regards,
> Ayan Guha
>
>
>
> ------------------------------
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 158 Ter Rue du Temple 75003 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à
> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
> destinataire de ce message, merci de le détruire et d'en avertir
> l'expéditeur.
>
--
Best Regards,
Ayan Guha
Re: Inode for STS
Posted by Christophe Préaud <ch...@kelkoo.com>.
Hi Ayan,
I have opened a JIRA about this issues, but there are no answer so far: SPARK-15401<https://issues.apache.org/jira/browse/SPARK-15401>
Regards,
Christophe.
On 13/07/16 05:54, ayan guha wrote:
Hi
We are running Spark Thrift Server as a long running application. However, it looks like it is filling up /tmp/hive folder with lots of small files and directories with no file in them, blowing out inode limit and preventing any connection with "No Space Left in Device" issue.
What is the best way to clean up those small files periodically?
--
Best Regards,
Ayan Guha
________________________________
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris
Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.