Creating bundled jar files for running under hadoop

[Andy Doddington <an...@doddington.net>]

I am trying to create a bundled jar file for running using the hadoop ‘jar’ command. However, when I try to do this it fails to
find the jar files and other resources that I have placed into the jar (pointed at by the Class-Path property in the MANIFEST.MF).

I have unpacked the jar file into a temporary directory and run it manually using the java -jar command (after manually listing
the various hadoop jar files that are required in the classpath) and it works fine in this mode.

I have read on some places that the workaround is to put my jar files and other resources into the hadoop ‘lib’ directory, but
I really don’t want to do this, since it feels horribly kludgy. I don’t mind creating a ‘lib’ directory in my jar file, since this will be
hidden, but I would appreciate more documentation as to why/how I need to do this.

Can anybody advise as to how I can get 'hadoop jar’ to honour the Class-Path entry in the MANIFEST.MF?

Thanks,

	Andy Doddington

Re: Creating bundled jar files for running under hadoop

[Steve Loughran <st...@apache.org>]

On 22/03/11 13:34, Andy Doddington wrote:
> I am trying to create a bundled jar file for running using the hadoop ‘jar’ command. However, when I try to do this it fails to
> find the jar files and other resources that I have placed into the jar (pointed at by the Class-Path property in the MANIFEST.MF).
>
> I have unpacked the jar file into a temporary directory and run it manually using the java -jar command (after manually listing
> the various hadoop jar files that are required in the classpath) and it works fine in this mode.
>
> I have read on some places that the workaround is to put my jar files and other resources into the hadoop ‘lib’ directory, but
> I really don’t want to do this, since it feels horribly kludgy.

Putting them in lib/ is bad as you need to restart the cluster to get 
changes out, and you can't have jobs with different versions in them

use the -libjars option instead

>I don’t mind creating a ‘lib’ directory in my jar file, since this will be
> hidden, but I would appreciate more documentation as to why/how I need to do this.

because the class-path manifest thing is designed for classloaders in 
the local filesystem where relative paths are used to find JARs and 
everything runs locally, and is only handled by the normal Java main 
entry point and the applet loader.

>
> Can anybody advise as to how I can get 'hadoop jar’ to honour the Class-Path entry in the MANIFEST.MF?

You'd probably have to write the code to
  -parse the entry at job submission
  -find the JARs or fail
  -somehow include all these JARs in the list of dependencies for the 
job submission list via a transform to the -libjars command.
  -add the tests for this

I'm not sure it's worth the effort.