You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "S.L" <si...@gmail.com> on 2014/04/29 04:42:32 UTC
hadoop.tmp.dir directory size
Hi Folks,
I am running a map only (no reduce) job on Hadoop 2.3.0 and the
/tmp/hadoop-df/nm-local-dir (df being my user name) directory is growing
exponentially causing me run out of my 80GB disk space.
I would like to know if there is any mechanism(automatic or manual) by
which I can free up the space in /tmp/hadoop-df/nm-local-dir while the job
is still running.
Thanks.
Re: hadoop.tmp.dir directory size
Posted by unmesha sreeveni <un...@gmail.com>.
Can You just try and see this.
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
fs.deleteOnExit("path/to/tmp");
On Thu, May 1, 2014 at 12:10 AM, S.L <si...@gmail.com> wrote:
> Can I do this while the job is still running ? I know I cant delete the
> directory , but I just want confirmation that the data Hadoop writes into
> /tmp/hadoop-df/nm-local-dir (df being my user name) can be discarded while
> the job is being executed.
>
>
> On Wed, Apr 30, 2014 at 6:40 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
>>
>> Try
>>
>> * hadoop fs -rmr /tmp*
>>
>> --
>> *Thanks & Regards *
>>
>>
>> *Unmesha Sreeveni U.B*
>> *Hadoop, Bigdata Developer*
>> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
>> http://www.unmeshasreeveni.blogspot.in/
>>
>>
>>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: hadoop.tmp.dir directory size
Posted by unmesha sreeveni <un...@gmail.com>.
Can You just try and see this.
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
fs.deleteOnExit("path/to/tmp");
On Thu, May 1, 2014 at 12:10 AM, S.L <si...@gmail.com> wrote:
> Can I do this while the job is still running ? I know I cant delete the
> directory , but I just want confirmation that the data Hadoop writes into
> /tmp/hadoop-df/nm-local-dir (df being my user name) can be discarded while
> the job is being executed.
>
>
> On Wed, Apr 30, 2014 at 6:40 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
>>
>> Try
>>
>> * hadoop fs -rmr /tmp*
>>
>> --
>> *Thanks & Regards *
>>
>>
>> *Unmesha Sreeveni U.B*
>> *Hadoop, Bigdata Developer*
>> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
>> http://www.unmeshasreeveni.blogspot.in/
>>
>>
>>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: hadoop.tmp.dir directory size
Posted by unmesha sreeveni <un...@gmail.com>.
Can You just try and see this.
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
fs.deleteOnExit("path/to/tmp");
On Thu, May 1, 2014 at 12:10 AM, S.L <si...@gmail.com> wrote:
> Can I do this while the job is still running ? I know I cant delete the
> directory , but I just want confirmation that the data Hadoop writes into
> /tmp/hadoop-df/nm-local-dir (df being my user name) can be discarded while
> the job is being executed.
>
>
> On Wed, Apr 30, 2014 at 6:40 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
>>
>> Try
>>
>> * hadoop fs -rmr /tmp*
>>
>> --
>> *Thanks & Regards *
>>
>>
>> *Unmesha Sreeveni U.B*
>> *Hadoop, Bigdata Developer*
>> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
>> http://www.unmeshasreeveni.blogspot.in/
>>
>>
>>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: hadoop.tmp.dir directory size
Posted by unmesha sreeveni <un...@gmail.com>.
Can You just try and see this.
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
fs.deleteOnExit("path/to/tmp");
On Thu, May 1, 2014 at 12:10 AM, S.L <si...@gmail.com> wrote:
> Can I do this while the job is still running ? I know I cant delete the
> directory , but I just want confirmation that the data Hadoop writes into
> /tmp/hadoop-df/nm-local-dir (df being my user name) can be discarded while
> the job is being executed.
>
>
> On Wed, Apr 30, 2014 at 6:40 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
>>
>> Try
>>
>> * hadoop fs -rmr /tmp*
>>
>> --
>> *Thanks & Regards *
>>
>>
>> *Unmesha Sreeveni U.B*
>> *Hadoop, Bigdata Developer*
>> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
>> http://www.unmeshasreeveni.blogspot.in/
>>
>>
>>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: hadoop.tmp.dir directory size
Posted by "S.L" <si...@gmail.com>.
Can I do this while the job is still running ? I know I cant delete the
directory , but I just want confirmation that the data Hadoop writes into
/tmp/hadoop-df/nm-local-dir (df being my user name) can be discarded while
the job is being executed.
On Wed, Apr 30, 2014 at 6:40 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
> Try
>
> *hadoop fs -rmr /tmp*
>
> --
> *Thanks & Regards *
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>
Re: hadoop.tmp.dir directory size
Posted by "S.L" <si...@gmail.com>.
Can I do this while the job is still running ? I know I cant delete the
directory , but I just want confirmation that the data Hadoop writes into
/tmp/hadoop-df/nm-local-dir (df being my user name) can be discarded while
the job is being executed.
On Wed, Apr 30, 2014 at 6:40 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
> Try
>
> *hadoop fs -rmr /tmp*
>
> --
> *Thanks & Regards *
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>
Re: hadoop.tmp.dir directory size
Posted by "S.L" <si...@gmail.com>.
Can I do this while the job is still running ? I know I cant delete the
directory , but I just want confirmation that the data Hadoop writes into
/tmp/hadoop-df/nm-local-dir (df being my user name) can be discarded while
the job is being executed.
On Wed, Apr 30, 2014 at 6:40 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
> Try
>
> *hadoop fs -rmr /tmp*
>
> --
> *Thanks & Regards *
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>
Re: hadoop.tmp.dir directory size
Posted by "S.L" <si...@gmail.com>.
Can I do this while the job is still running ? I know I cant delete the
directory , but I just want confirmation that the data Hadoop writes into
/tmp/hadoop-df/nm-local-dir (df being my user name) can be discarded while
the job is being executed.
On Wed, Apr 30, 2014 at 6:40 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
> Try
>
> *hadoop fs -rmr /tmp*
>
> --
> *Thanks & Regards *
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>
Re: hadoop.tmp.dir directory size
Posted by unmesha sreeveni <un...@gmail.com>.
Try
*hadoop fs -rmr /tmp*
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: hadoop.tmp.dir directory size
Posted by unmesha sreeveni <un...@gmail.com>.
Try
*hadoop fs -rmr /tmp*
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: hadoop.tmp.dir directory size
Posted by unmesha sreeveni <un...@gmail.com>.
Try
*hadoop fs -rmr /tmp*
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: hadoop.tmp.dir directory size
Posted by unmesha sreeveni <un...@gmail.com>.
Try
*hadoop fs -rmr /tmp*
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/