You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Brian C. Huffman" <bh...@etinternational.com> on 2014/02/21 15:03:14 UTC

Cleanup after Yarn Job

All,

I'm trying to model a Yarn Client after the Distributed Shell example.  
However I'd like to add a method to cleanup the job's files after 
completion.

I've defined a cleanup routine:
   private void cleanup(ApplicationId appId, FileSystem fs)
       throws IOException {
     String PathSuffix = appName + "/" + appId.getId();
     Path Dst = new Path(fs.getHomeDirectory(), PathSuffix);
     fs.delete(Dst, true);
   }

The problem that I'm having is that I'd like to call it after 
monitorApplication exits, but in the case that the time limit was 
exceeded and killApplication is called, both the appId and the 
FileSystem objects are gone.  I could get around the appId issue since I 
really only need a String or integer representation, but since Yarn 
Client seems to be managing the filesystem object (the example uses 
FileSystem.get(conf)), I'm not sure of a way around that unless I create 
my own FileSystem object.

Any suggestions?

Thanks,
Brian