nutch vs hadoop package versions

[Luke Mawbey <ju...@lbm.net.au>]

Dear all,

I'm using Nutch 1.7 with Hadoop 1.2. Both use commons-lang.jar, but 
Nutch uses version 2.6 and Hadoop uses version 2.4. I have a plugin that 
needs some features available in 2.6 though not in 2.4. When I run in 
standalone mode, everything is fine, when I run on Hadoop, the extra 
functionality of 2.6 is not available.

The brute force method of getting this to work appears to be to download 
the source for Hadoop, change the version of commons-lang and recompile. 
Is there a more elegant solution?

Apologies in advance if this is a fairly basic ivy/ant/java question.

Regards,
Luke.

Re: nutch vs hadoop package versions

[feng lu <am...@gmail.com>]

Hi

Do you use Nutch ant command to compile and package Nutch and your plugin
source code. It will generate a packaged job file like
this apache-nutch-1.8-SNAPSHOT.job that include all third-party libraries
and all plugins jar. So it can be run on Hadoop cluster.




On Mon, Mar 3, 2014 at 8:37 PM, Luke Mawbey <ju...@lbm.net.au> wrote:

> Dear all,
>
> I'm using Nutch 1.7 with Hadoop 1.2. Both use commons-lang.jar, but Nutch
> uses version 2.6 and Hadoop uses version 2.4. I have a plugin that needs
> some features available in 2.6 though not in 2.4. When I run in standalone
> mode, everything is fine, when I run on Hadoop, the extra functionality of
> 2.6 is not available.
>
> The brute force method of getting this to work appears to be to download
> the source for Hadoop, change the version of commons-lang and recompile. Is
> there a more elegant solution?
>
> Apologies in advance if this is a fairly basic ivy/ant/java question.
>
> Regards,
> Luke.
>
>


-- 
Don't Grow Old, Grow Up... :-)

Re: nutch vs hadoop package versions

["S.L" <si...@gmail.com>]

Hi,

There is a property to set the precedence for the jars in the submitted
MapReduce job over the Hadoop jars, I dont recall the name of the property
but you can try googling it.


On Mon, Mar 3, 2014 at 7:37 AM, Luke Mawbey <ju...@lbm.net.au> wrote:

> Dear all,
>
> I'm using Nutch 1.7 with Hadoop 1.2. Both use commons-lang.jar, but Nutch
> uses version 2.6 and Hadoop uses version 2.4. I have a plugin that needs
> some features available in 2.6 though not in 2.4. When I run in standalone
> mode, everything is fine, when I run on Hadoop, the extra functionality of
> 2.6 is not available.
>
> The brute force method of getting this to work appears to be to download
> the source for Hadoop, change the version of commons-lang and recompile. Is
> there a more elegant solution?
>
> Apologies in advance if this is a fairly basic ivy/ant/java question.
>
> Regards,
> Luke.
>
>