You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by #YONG YONG CHENG# <aa...@pmail.ntu.edu.sg> on 2009/09/03 05:20:02 UTC

DistributedCache purgeCache()

Good Day,

I have a question on the DistributedCache as follows.

I have used DistributedCache to move my executable(.exe) around the (onto the local filesystems of) nodes in Hadoop and run the .exe (via addCacheArchive() and getLocalCacheArchives()). But I discovered after my job, the .exe still resides on the nodes. I have try using releaseCache() and/or purgeCache(), but the .exe still on the nodes. Is there a way to remove the .exe from the nodes?

Thanks.

Re: DistributedCache purgeCache()

Posted by Vinod KV <vi...@yahoo-inc.com>.

It is not expected that the users clear the distributed cache files. The 
files are left as they are on the TT so that they can be used by other 
jobs that refer to the same cache files on the same TT. They will be 
deleted by the TT only when it is shutting down or when it tries to 
purge some unused files when the total disk usage of all dist-cache 
files hits a limit.

HTH,
+Vinod

Amogh Vasekar wrote:
> My bad, its not exposed. 
> If nothing else's available, may be mimic its behaviour in the code?
>
> Thanks,
> Amogh 
>
> -----Original Message-----
> From: #YONG YONG CHENG# [mailto:aarnchng@pmail.ntu.edu.sg] 
> Sent: Monday, September 07, 2009 10:27 AM
> To: common-user@hadoop.apache.org
> Subject: RE: DistributedCache purgeCache()
>
> Thanks for your swift response.
>
> But where can I find deletecache()?
>
> Thanks.
>
>
> -----Original Message-----
> From: Amogh Vasekar [mailto:amogh@yahoo-inc.com]
> Sent: Thu 9/3/2009 2:44 PM
> To: common-user@hadoop.apache.org
> Subject: RE: DistributedCache purgeCache()
>  
> AFAIK, releaseCache only works on cleaning reference to your file. Try using deletecache in synchronized manner.
>
> Thanks,
> Amogh
>
> -----Original Message-----
> From: #YONG YONG CHENG# [mailto:aarnchng@pmail.ntu.edu.sg] 
> Sent: Thursday, September 03, 2009 8:50 AM
> To: common-user@hadoop.apache.org
> Subject: DistributedCache purgeCache()
>
> Good Day,
>
> I have a question on the DistributedCache as follows.
>
> I have used DistributedCache to move my executable(.exe) around the (onto the local filesystems of) nodes in Hadoop and run the .exe (via addCacheArchive() and getLocalCacheArchives()). But I discovered after my job, the .exe still resides on the nodes. I have try using releaseCache() and/or purgeCache(), but the .exe still on the nodes. Is there a way to remove the .exe from the nodes?
>
> Thanks.
>
>
>

RE: DistributedCache purgeCache()

Posted by Amogh Vasekar <am...@yahoo-inc.com>.

My bad, its not exposed. 
If nothing else's available, may be mimic its behaviour in the code?

Thanks,
Amogh 

-----Original Message-----
From: #YONG YONG CHENG# [mailto:aarnchng@pmail.ntu.edu.sg] 
Sent: Monday, September 07, 2009 10:27 AM
To: common-user@hadoop.apache.org
Subject: RE: DistributedCache purgeCache()

Thanks for your swift response.

But where can I find deletecache()?

Thanks.

-----Original Message-----
From: Amogh Vasekar [mailto:amogh@yahoo-inc.com]
Sent: Thu 9/3/2009 2:44 PM
To: common-user@hadoop.apache.org
Subject: RE: DistributedCache purgeCache()

AFAIK, releaseCache only works on cleaning reference to your file. Try using deletecache in synchronized manner.

Thanks,
Amogh

-----Original Message-----
From: #YONG YONG CHENG# [mailto:aarnchng@pmail.ntu.edu.sg] 
Sent: Thursday, September 03, 2009 8:50 AM
To: common-user@hadoop.apache.org
Subject: DistributedCache purgeCache()

Good Day,

I have a question on the DistributedCache as follows.

I have used DistributedCache to move my executable(.exe) around the (onto the local filesystems of) nodes in Hadoop and run the .exe (via addCacheArchive() and getLocalCacheArchives()). But I discovered after my job, the .exe still resides on the nodes. I have try using releaseCache() and/or purgeCache(), but the .exe still on the nodes. Is there a way to remove the .exe from the nodes?

Thanks.

RE: DistributedCache purgeCache()

Posted by #YONG YONG CHENG# <aa...@pmail.ntu.edu.sg>.

Thanks for your swift response.

But where can I find deletecache()?

Thanks.


-----Original Message-----
From: Amogh Vasekar [mailto:amogh@yahoo-inc.com]
Sent: Thu 9/3/2009 2:44 PM
To: common-user@hadoop.apache.org
Subject: RE: DistributedCache purgeCache()
 
AFAIK, releaseCache only works on cleaning reference to your file. Try using deletecache in synchronized manner.

Thanks,
Amogh

-----Original Message-----
From: #YONG YONG CHENG# [mailto:aarnchng@pmail.ntu.edu.sg] 
Sent: Thursday, September 03, 2009 8:50 AM
To: common-user@hadoop.apache.org
Subject: DistributedCache purgeCache()

Good Day,

I have a question on the DistributedCache as follows.

I have used DistributedCache to move my executable(.exe) around the (onto the local filesystems of) nodes in Hadoop and run the .exe (via addCacheArchive() and getLocalCacheArchives()). But I discovered after my job, the .exe still resides on the nodes. I have try using releaseCache() and/or purgeCache(), but the .exe still on the nodes. Is there a way to remove the .exe from the nodes?

Thanks.

RE: DistributedCache purgeCache()

Posted by Amogh Vasekar <am...@yahoo-inc.com>.

AFAIK, releaseCache only works on cleaning reference to your file. Try using deletecache in synchronized manner.

Thanks,
Amogh

-----Original Message-----
From: #YONG YONG CHENG# [mailto:aarnchng@pmail.ntu.edu.sg] 
Sent: Thursday, September 03, 2009 8:50 AM
To: common-user@hadoop.apache.org
Subject: DistributedCache purgeCache()

Good Day,

I have a question on the DistributedCache as follows.

I have used DistributedCache to move my executable(.exe) around the (onto the local filesystems of) nodes in Hadoop and run the .exe (via addCacheArchive() and getLocalCacheArchives()). But I discovered after my job, the .exe still resides on the nodes. I have try using releaseCache() and/or purgeCache(), but the .exe still on the nodes. Is there a way to remove the .exe from the nodes?

Thanks.