You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sky <sk...@hotmail.com> on 2012/04/09 07:44:59 UTC
How do I include the newer version of Commons-lang in my jar?
Hi.
I am new to Hadoop and I am working on project on AWS Elastic MapReduce.
The problem I am facing is:
* org.apache.commons.lang.time.DateUtils: parseDate() works OK but
parseDateStrictly() fails.
I think parseDateStrictly might be new in lang 2.5. I thought I included all
dependencies. However, for some reason, during runtime, my app is not
picking up the newer commons-lang.
Would love some help.
Thx
- sky
Re: How do I include the newer version of Commons-lang in my jar?
Posted by Harsh J <ha...@cloudera.com>.
Sky,
Yes you are right in your summary. Thanks also for reporting back on
the 0.205/1.x issue.
One ugly hack that just comes to mind, would be to manipulate the
classloader at runtime. I've not tried it out to know for sure if its
possible/will work, but just thought I'd note it down.
However, your statement "then, there is not possible for me to use an
external jar such as "commons-lang" from apache in my application. Any
external jars packaged within my jar under "lib" directory are not
captured." needs some refinement - You can plug and send dependent
jars to a job for sure - but if that jar is already present installed
on the cluster (i.e., hadoop uses it via its $HADOOP_HOME/lib/ paths),
then you can't override it unless you work around.
On Tue, Apr 10, 2012 at 2:16 AM, Sky USC <sk...@hotmail.com> wrote:
>
>
>
>
> Thanks for the reply. I appreciate your helpfulness. I created Jars by following instructions at "http://blog.mafr.de/2010/07/24/maven-hadoop-job/". So external Jars are stored in lib/ folder within a jar.
>
> Am I summarizing this correctly:
> 1. If hadoop version = 0.20.203 or lower - then, there is not possible for me to use an external jar such as "commons-lang" from apache in my application. Any external jars packaged within my jar under "lib" directory are not captured. This appears like a huge limitation to me?
> 2. If hadoop version > 0.20.204 to 1.0.x - then use "HADOOP_USER_CLASSPATH_FIRST=true" environment variable before launching "hadoop jar" might help. I tried this for version 0.20.205 but it didnt work.
> 3. If hadoop version > 2.x or formerly 0.23.x - then this can be set via API?
>
> Is there a working version of testable jar that has these dependencies that I can try to figure out if its my way of packaging jar or something else??
>
> Thx
>
>> From: harsh@cloudera.com
>> Date: Mon, 9 Apr 2012 13:50:37 +0530
>> Subject: Re: How do I include the newer version of Commons-lang in my jar?
>> To: common-user@hadoop.apache.org
>>
>> Answer is a bit messy.
>>
>> Perhaps you can set the environment variable "export
>> HADOOP_USER_CLASSPATH_FIRST=true" before you do a "hadoop jar …" to
>> launch your job. However, although this approach is present in
>> 0.20.204+ (0.20.205, and 1.0.x), am not sure if it makes an impact on
>> the tasks as well. I don't see it changing anything but for the driver
>> CP. I've not tested it - please let us know if it works in your
>> environment.
>>
>> In higher versions (2.x or formerly 0.23.x), this is doable from
>> within your job if you set "mapreduce.job.user.classpath.first" to
>> true inside your job, and ship your replacement jars along.
>>
>> Some versions would also let you set this via
>> "JobConf/Job.setUserClassesTakesPrecedence(true/false)" API calls.
>>
>> On Mon, Apr 9, 2012 at 11:14 AM, Sky <sk...@hotmail.com> wrote:
>> > Hi.
>> >
>> > I am new to Hadoop and I am working on project on AWS Elastic MapReduce.
>> >
>> > The problem I am facing is:
>> > * org.apache.commons.lang.time.DateUtils: parseDate() works OK but
>> > parseDateStrictly() fails.
>> > I think parseDateStrictly might be new in lang 2.5. I thought I included all
>> > dependencies. However, for some reason, during runtime, my app is not
>> > picking up the newer commons-lang.
>> >
>> > Would love some help.
>> >
>> > Thx
>> > - sky
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: How do I include the newer version of Commons-lang in my jar?
Posted by John George <jo...@yahoo-inc.com>.
Have you tried setting 'mapreduce.user.classpath.first'? It allows user
jars to be put in the classpath before hadoop jars.
-----Original Message-----
From: Sky USC <sk...@hotmail.com>
Reply-To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>
Date: Mon, 9 Apr 2012 15:46:52 -0500
To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>
Subject: RE: How do I include the newer version of Commons-lang in my jar?
>
>
>
>
>Thanks for the reply. I appreciate your helpfulness. I created Jars by
>following instructions at
>"http://blog.mafr.de/2010/07/24/maven-hadoop-job/". So external Jars are
>stored in lib/ folder within a jar.
>
>Am I summarizing this correctly:
>1. If hadoop version = 0.20.203 or lower - then, there is not possible
>for me to use an external jar such as "commons-lang" from apache in my
>application. Any external jars packaged within my jar under "lib"
>directory are not captured. This appears like a huge limitation to me?
>2. If hadoop version > 0.20.204 to 1.0.x - then use
>"HADOOP_USER_CLASSPATH_FIRST=true" environment variable before launching
>"hadoop jar" might help. I tried this for version 0.20.205 but it didnt
>work.
>3. If hadoop version > 2.x or formerly 0.23.x - then this can be set via
>API?
>
>Is there a working version of testable jar that has these dependencies
>that I can try to figure out if its my way of packaging jar or something
>else??
>
>Thx
>
>> From: harsh@cloudera.com
>> Date: Mon, 9 Apr 2012 13:50:37 +0530
>> Subject: Re: How do I include the newer version of Commons-lang in my
>>jar?
>> To: common-user@hadoop.apache.org
>>
>> Answer is a bit messy.
>>
>> Perhaps you can set the environment variable "export
>> HADOOP_USER_CLASSPATH_FIRST=true" before you do a "hadoop jar Š" to
>> launch your job. However, although this approach is present in
>> 0.20.204+ (0.20.205, and 1.0.x), am not sure if it makes an impact on
>> the tasks as well. I don't see it changing anything but for the driver
>> CP. I've not tested it - please let us know if it works in your
>> environment.
>>
>> In higher versions (2.x or formerly 0.23.x), this is doable from
>> within your job if you set "mapreduce.job.user.classpath.first" to
>> true inside your job, and ship your replacement jars along.
>>
>> Some versions would also let you set this via
>> "JobConf/Job.setUserClassesTakesPrecedence(true/false)" API calls.
>>
>> On Mon, Apr 9, 2012 at 11:14 AM, Sky <sk...@hotmail.com> wrote:
>> > Hi.
>> >
>> > I am new to Hadoop and I am working on project on AWS Elastic
>>MapReduce.
>> >
>> > The problem I am facing is:
>> > * org.apache.commons.lang.time.DateUtils: parseDate() works OK but
>> > parseDateStrictly() fails.
>> > I think parseDateStrictly might be new in lang 2.5. I thought I
>>included all
>> > dependencies. However, for some reason, during runtime, my app is not
>> > picking up the newer commons-lang.
>> >
>> > Would love some help.
>> >
>> > Thx
>> > - sky
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
RE: How do I include the newer version of Commons-lang in my jar?
Posted by Sky USC <sk...@hotmail.com>.
Thanks for the reply. I appreciate your helpfulness. I created Jars by following instructions at "http://blog.mafr.de/2010/07/24/maven-hadoop-job/". So external Jars are stored in lib/ folder within a jar.
Am I summarizing this correctly:
1. If hadoop version = 0.20.203 or lower - then, there is not possible for me to use an external jar such as "commons-lang" from apache in my application. Any external jars packaged within my jar under "lib" directory are not captured. This appears like a huge limitation to me?
2. If hadoop version > 0.20.204 to 1.0.x - then use "HADOOP_USER_CLASSPATH_FIRST=true" environment variable before launching "hadoop jar" might help. I tried this for version 0.20.205 but it didnt work.
3. If hadoop version > 2.x or formerly 0.23.x - then this can be set via API?
Is there a working version of testable jar that has these dependencies that I can try to figure out if its my way of packaging jar or something else??
Thx
> From: harsh@cloudera.com
> Date: Mon, 9 Apr 2012 13:50:37 +0530
> Subject: Re: How do I include the newer version of Commons-lang in my jar?
> To: common-user@hadoop.apache.org
>
> Answer is a bit messy.
>
> Perhaps you can set the environment variable "export
> HADOOP_USER_CLASSPATH_FIRST=true" before you do a "hadoop jar …" to
> launch your job. However, although this approach is present in
> 0.20.204+ (0.20.205, and 1.0.x), am not sure if it makes an impact on
> the tasks as well. I don't see it changing anything but for the driver
> CP. I've not tested it - please let us know if it works in your
> environment.
>
> In higher versions (2.x or formerly 0.23.x), this is doable from
> within your job if you set "mapreduce.job.user.classpath.first" to
> true inside your job, and ship your replacement jars along.
>
> Some versions would also let you set this via
> "JobConf/Job.setUserClassesTakesPrecedence(true/false)" API calls.
>
> On Mon, Apr 9, 2012 at 11:14 AM, Sky <sk...@hotmail.com> wrote:
> > Hi.
> >
> > I am new to Hadoop and I am working on project on AWS Elastic MapReduce.
> >
> > The problem I am facing is:
> > * org.apache.commons.lang.time.DateUtils: parseDate() works OK but
> > parseDateStrictly() fails.
> > I think parseDateStrictly might be new in lang 2.5. I thought I included all
> > dependencies. However, for some reason, during runtime, my app is not
> > picking up the newer commons-lang.
> >
> > Would love some help.
> >
> > Thx
> > - sky
> >
> >
>
>
>
> --
> Harsh J
Re: How do I include the newer version of Commons-lang in my jar?
Posted by Harsh J <ha...@cloudera.com>.
Answer is a bit messy.
Perhaps you can set the environment variable "export
HADOOP_USER_CLASSPATH_FIRST=true" before you do a "hadoop jar …" to
launch your job. However, although this approach is present in
0.20.204+ (0.20.205, and 1.0.x), am not sure if it makes an impact on
the tasks as well. I don't see it changing anything but for the driver
CP. I've not tested it - please let us know if it works in your
environment.
In higher versions (2.x or formerly 0.23.x), this is doable from
within your job if you set "mapreduce.job.user.classpath.first" to
true inside your job, and ship your replacement jars along.
Some versions would also let you set this via
"JobConf/Job.setUserClassesTakesPrecedence(true/false)" API calls.
On Mon, Apr 9, 2012 at 11:14 AM, Sky <sk...@hotmail.com> wrote:
> Hi.
>
> I am new to Hadoop and I am working on project on AWS Elastic MapReduce.
>
> The problem I am facing is:
> * org.apache.commons.lang.time.DateUtils: parseDate() works OK but
> parseDateStrictly() fails.
> I think parseDateStrictly might be new in lang 2.5. I thought I included all
> dependencies. However, for some reason, during runtime, my app is not
> picking up the newer commons-lang.
>
> Would love some help.
>
> Thx
> - sky
>
>
--
Harsh J