RE: How to make PIG delete its temporary files ?

[rakesh kothari <rk...@hotmail.com>]

This is is happening on PIG 0.7 as well. When I use PigServer to launch pig jobs from Java lots of tmp files (e.g. directory /tmp/temp-203305867) gets created that are not deleted at the end of the Job.

"hadoop.tmp.dir" points to some other directory "/tmp/hadoop-userA". Any ideas on how to get path to these tmp folders so that I can explicitly delete them at the end of the Job ?

Thanks,
-Rakesh

> Date: Mon, 25 Oct 2010 20:30:24 +0200
> From: vbarat@ubikod.com
> To: user@pig.apache.org
> Subject: Re: How to make PIG delete its temporary files ?
> 
> Thanks for your answer, I will investigate on that.
> 
> Le 25/10/10 08:26, Rekha Joshi a écrit :
> > Hi Vincent,
> >
> > I recall it happens under some specific condition and is resolved in later versions of pig/hadoop by adding an explicit cleanup before final exit.
> > As a workaround in 0.6 you can code finally{} in your java file to remove files from your user specific folder under hadoop.tmp.dir.HTH!
> >
> > Thanks&  Regards,
> > /Rekha.
> >
> > On 10/24/10 5:24 AM, "Vincent Barat"<vb...@ubikod.com>  wrote:
> >
> > I forgot to mention that I use PIG 0.6.0, it may help !
> >
> > Le 23/10/10 13:29, Vincent Barat a écrit :
> >> Hello,
> >>
> >> I face an issue with PIG temporary files: they are not deleted
> >> once a job is terminated.
> >> I got my HDFS storage full of PIG temporary files.
> >>
> >> I use PIG from Java using a PigServer object.
> >>
> >> Is there is way to ask PIG to cleanup all that it created to
> >> perform a job ?
> >>
> >> Thanks a lot
> >
 		 	   		  

Re: How to make PIG delete its temporary files ?

[Alan Gates <ga...@yahoo-inc.com>]

Pig is supposed to remove all these temporary files, as long as the  
java process finishes in such a way that it has a chance to clean up  
(ie, no one does a kill -9 on it or something).  Can you file a JIRA  
with a reproducible case so we can track this down and fix it?

Alan.

On Nov 22, 2010, at 2:20 PM, rakesh kothari wrote:

>
> This is is happening on PIG 0.7 as well. When I use PigServer to  
> launch pig jobs from Java lots of tmp files (e.g. directory /tmp/ 
> temp-203305867) gets created that are not deleted at the end of the  
> Job.
>
> "hadoop.tmp.dir" points to some other directory "/tmp/hadoop-userA".  
> Any ideas on how to get path to these tmp folders so that I can  
> explicitly delete them at the end of the Job ?
>
> Thanks,
> -Rakesh
>
>> Date: Mon, 25 Oct 2010 20:30:24 +0200
>> From: vbarat@ubikod.com
>> To: user@pig.apache.org
>> Subject: Re: How to make PIG delete its temporary files ?
>>
>> Thanks for your answer, I will investigate on that.
>>
>> Le 25/10/10 08:26, Rekha Joshi a écrit :
>>> Hi Vincent,
>>>
>>> I recall it happens under some specific condition and is resolved  
>>> in later versions of pig/hadoop by adding an explicit cleanup  
>>> before final exit.
>>> As a workaround in 0.6 you can code finally{} in your java file to  
>>> remove files from your user specific folder under  
>>> hadoop.tmp.dir.HTH!
>>>
>>> Thanks&  Regards,
>>> /Rekha.
>>>
>>> On 10/24/10 5:24 AM, "Vincent Barat"<vb...@ubikod.com>  wrote:
>>>
>>> I forgot to mention that I use PIG 0.6.0, it may help !
>>>
>>> Le 23/10/10 13:29, Vincent Barat a écrit :
>>>> Hello,
>>>>
>>>> I face an issue with PIG temporary files: they are not deleted
>>>> once a job is terminated.
>>>> I got my HDFS storage full of PIG temporary files.
>>>>
>>>> I use PIG from Java using a PigServer object.
>>>>
>>>> Is there is way to ask PIG to cleanup all that it created to
>>>> perform a job ?
>>>>
>>>> Thanks a lot
>>>
>