MapReduce and Jar Files

[Albert Chern <al...@gmail.com>]

Hello,

I have run into a problem when attempting to run jobs that require
code from multiple jars.  The JobConf only allows me to set one jar
file, so the other jars are not copied to the task trackers, and hence
the jobs fail due to the dependencies.  Besides merging the jars or
manually copying them to each machine, is there a solution for this?

Thanks in advance,
Albert

Re: MapReduce and Jar Files

[Darek Zbik <d....@softwaremind.pl>]

Albert Chern wrote:
> Hello,
>
> I have run into a problem when attempting to run jobs that require
> code from multiple jars.  The JobConf only allows me to set one jar
> file, so the other jars are not copied to the task trackers, and hence
> the jobs fail due to the dependencies.  Besides merging the jars or
> manually copying them to each machine, is there a solution for this? 

Try to create lib directory in your jar and put your dependency into 
this directory. Contents of the jar's lib directory is added to 
classloader during starting map and reduce tasks. Unfortunately it does 
not work if you try to use additional libraries in InputReader in this 
case classlaoder is not modified.

Retards,

darek