Deep Magic on the distributed classpath

[John Armstrong <jo...@ccri.com>]

So I think I've figured out how to fix my problem with putting files on
the distributed classpath by digging through the code Hadoop uses to
process -libjars.

If I say

DistributedCache.addFileToClassPath(hdfsFile,conf);

then hdfsFile is added to the distributed cache, but doesn't show upon the
classpath of the mappers or reducers, leading to crashings and burnings.

But if I say

DistributedCache.addFileToClassPath(new
Path(hdfsFile.toUri().getPath()),conf);

then hdfsFile *does* show up on the classpath, and my jobs run fine.

So here's the thing: if I grab the code at this point in a debugger and
check

new Path(hdfsFile.toUri().getPath()).equals(hdfsFile)

I *always* get true.  That is, according to Path.equals(), hdfsFile and
new Path(hdfsFile.toUri().getPath()) are semantically identical.

So while I'm glad that I've got something working (and hopefully this will
help anyone stuck in a similar place in the future), I'm incredibly
confused as to *why* it works.  Any ideas?

Re: Deep Magic on the distributed classpath

[John Armstrong <jo...@ccri.com>]

On Wed, 27 Jul 2011 10:58:17 -0400, David Rosenstrauch <da...@darose.net>
wrote:
> There is another, easier approach:  if your app inherits from the Tool 
> class / runs via ToolRunner, then your app can inherit the -libjars 
> command line functionality itself.

This is true; the problem with this approach is that we're also trying to
use cache persistence to our advantage.  Using libjars makes a new HDFS
copy (somewhere under /tmp) of the library every time I submit the job from
the command line, which has a new timestamp.  When the nodes check the
cache they see that their old copy of the library has an older timestamp
than the new one (despite the fact that they're actually the same file) and
asks for a new copy of the library for the local cache.

The upshot is: using libjars makes at most one local copy per node per job
run, but it will always make a new local copy each job run.  Adding the
library programatically can use older copies that are still cached from
previous job runs, leading to even less network overhead.

Re: Deep Magic on the distributed classpath

[David Rosenstrauch <da...@darose.net>]

On 07/27/2011 10:39 AM, John Armstrong wrote:
> So I think I've figured out how to fix my problem with putting files on
> the distributed classpath by digging through the code Hadoop uses to
> process -libjars.

There is another, easier approach:  if your app inherits from the Tool 
class / runs via ToolRunner, then your app can inherit the -libjars 
command line functionality itself.

DR