You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Vincent Barat <vb...@ubikod.com> on 2010/10/23 13:29:27 UTC
How to make PIG delete its temporary files ?
Hello,
I face an issue with PIG temporary files: they are not deleted once
a job is terminated.
I got my HDFS storage full of PIG temporary files.
I use PIG from Java using a PigServer object.
Is there is way to ask PIG to cleanup all that it created to perform
a job ?
Thanks a lot
Re: How to make PIG delete its temporary files ?
Posted by Alan Gates <ga...@yahoo-inc.com>.
Pig is supposed to remove all these temporary files, as long as the
java process finishes in such a way that it has a chance to clean up
(ie, no one does a kill -9 on it or something). Can you file a JIRA
with a reproducible case so we can track this down and fix it?
Alan.
On Nov 22, 2010, at 2:20 PM, rakesh kothari wrote:
>
> This is is happening on PIG 0.7 as well. When I use PigServer to
> launch pig jobs from Java lots of tmp files (e.g. directory /tmp/
> temp-203305867) gets created that are not deleted at the end of the
> Job.
>
> "hadoop.tmp.dir" points to some other directory "/tmp/hadoop-userA".
> Any ideas on how to get path to these tmp folders so that I can
> explicitly delete them at the end of the Job ?
>
> Thanks,
> -Rakesh
>
>> Date: Mon, 25 Oct 2010 20:30:24 +0200
>> From: vbarat@ubikod.com
>> To: user@pig.apache.org
>> Subject: Re: How to make PIG delete its temporary files ?
>>
>> Thanks for your answer, I will investigate on that.
>>
>> Le 25/10/10 08:26, Rekha Joshi a écrit :
>>> Hi Vincent,
>>>
>>> I recall it happens under some specific condition and is resolved
>>> in later versions of pig/hadoop by adding an explicit cleanup
>>> before final exit.
>>> As a workaround in 0.6 you can code finally{} in your java file to
>>> remove files from your user specific folder under
>>> hadoop.tmp.dir.HTH!
>>>
>>> Thanks& Regards,
>>> /Rekha.
>>>
>>> On 10/24/10 5:24 AM, "Vincent Barat"<vb...@ubikod.com> wrote:
>>>
>>> I forgot to mention that I use PIG 0.6.0, it may help !
>>>
>>> Le 23/10/10 13:29, Vincent Barat a écrit :
>>>> Hello,
>>>>
>>>> I face an issue with PIG temporary files: they are not deleted
>>>> once a job is terminated.
>>>> I got my HDFS storage full of PIG temporary files.
>>>>
>>>> I use PIG from Java using a PigServer object.
>>>>
>>>> Is there is way to ask PIG to cleanup all that it created to
>>>> perform a job ?
>>>>
>>>> Thanks a lot
>>>
>
RE: How to make PIG delete its temporary files ?
Posted by rakesh kothari <rk...@hotmail.com>.
This is is happening on PIG 0.7 as well. When I use PigServer to launch pig jobs from Java lots of tmp files (e.g. directory /tmp/temp-203305867) gets created that are not deleted at the end of the Job.
"hadoop.tmp.dir" points to some other directory "/tmp/hadoop-userA". Any ideas on how to get path to these tmp folders so that I can explicitly delete them at the end of the Job ?
Thanks,
-Rakesh
> Date: Mon, 25 Oct 2010 20:30:24 +0200
> From: vbarat@ubikod.com
> To: user@pig.apache.org
> Subject: Re: How to make PIG delete its temporary files ?
>
> Thanks for your answer, I will investigate on that.
>
> Le 25/10/10 08:26, Rekha Joshi a écrit :
> > Hi Vincent,
> >
> > I recall it happens under some specific condition and is resolved in later versions of pig/hadoop by adding an explicit cleanup before final exit.
> > As a workaround in 0.6 you can code finally{} in your java file to remove files from your user specific folder under hadoop.tmp.dir.HTH!
> >
> > Thanks& Regards,
> > /Rekha.
> >
> > On 10/24/10 5:24 AM, "Vincent Barat"<vb...@ubikod.com> wrote:
> >
> > I forgot to mention that I use PIG 0.6.0, it may help !
> >
> > Le 23/10/10 13:29, Vincent Barat a écrit :
> >> Hello,
> >>
> >> I face an issue with PIG temporary files: they are not deleted
> >> once a job is terminated.
> >> I got my HDFS storage full of PIG temporary files.
> >>
> >> I use PIG from Java using a PigServer object.
> >>
> >> Is there is way to ask PIG to cleanup all that it created to
> >> perform a job ?
> >>
> >> Thanks a lot
> >
Re: How to make PIG delete its temporary files ?
Posted by Vincent Barat <vb...@ubikod.com>.
Thanks for your answer, I will investigate on that.
Le 25/10/10 08:26, Rekha Joshi a écrit :
> Hi Vincent,
>
> I recall it happens under some specific condition and is resolved in later versions of pig/hadoop by adding an explicit cleanup before final exit.
> As a workaround in 0.6 you can code finally{} in your java file to remove files from your user specific folder under hadoop.tmp.dir.HTH!
>
> Thanks& Regards,
> /Rekha.
>
> On 10/24/10 5:24 AM, "Vincent Barat"<vb...@ubikod.com> wrote:
>
> I forgot to mention that I use PIG 0.6.0, it may help !
>
> Le 23/10/10 13:29, Vincent Barat a écrit :
>> Hello,
>>
>> I face an issue with PIG temporary files: they are not deleted
>> once a job is terminated.
>> I got my HDFS storage full of PIG temporary files.
>>
>> I use PIG from Java using a PigServer object.
>>
>> Is there is way to ask PIG to cleanup all that it created to
>> perform a job ?
>>
>> Thanks a lot
>
Re: How to make PIG delete its temporary files ?
Posted by Rekha Joshi <re...@yahoo-inc.com>.
Hi Vincent,
I recall it happens under some specific condition and is resolved in later versions of pig/hadoop by adding an explicit cleanup before final exit.
As a workaround in 0.6 you can code finally{} in your java file to remove files from your user specific folder under hadoop.tmp.dir.HTH!
Thanks & Regards,
/Rekha.
On 10/24/10 5:24 AM, "Vincent Barat" <vb...@ubikod.com> wrote:
I forgot to mention that I use PIG 0.6.0, it may help !
Le 23/10/10 13:29, Vincent Barat a écrit :
> Hello,
>
> I face an issue with PIG temporary files: they are not deleted
> once a job is terminated.
> I got my HDFS storage full of PIG temporary files.
>
> I use PIG from Java using a PigServer object.
>
> Is there is way to ask PIG to cleanup all that it created to
> perform a job ?
>
> Thanks a lot
Re: How to make PIG delete its temporary files ?
Posted by Vincent Barat <vb...@ubikod.com>.
I forgot to mention that I use PIG 0.6.0, it may help !
Le 23/10/10 13:29, Vincent Barat a écrit :
> Hello,
>
> I face an issue with PIG temporary files: they are not deleted
> once a job is terminated.
> I got my HDFS storage full of PIG temporary files.
>
> I use PIG from Java using a PigServer object.
>
> Is there is way to ask PIG to cleanup all that it created to
> perform a job ?
>
> Thanks a lot