You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by dmaniar <dh...@gmail.com> on 2012/04/11 06:09:10 UTC
Hadoopp_ClassPath issue.
Hi,
I am new to hadoop and its not very familiar with internal working. I had
some questions about HADOOP_CLASSPATH.
We are currently suppose to use a Hadoop cluster with 4 machines and its
HADOOP_CLASSPATH in hadoop-env.sh is as below.
export
HADOOP_CLASSPATH="/home/user/app/www/WEB-INF/classes:$HADOOP_CLASSPATH"
Now my,
/home/user/app/www/WEB-INF/classes has a class called Application.class
>From a remote machine I submit a map-reduce job to this cluster, with a jar
called MyJar.jar. [This has a Application.class too, but with some
modifications]
When the TaskTracker spawns a child Java process for the Mapper the
classpath I see is as below in that order,
Lets say my hadoop is installed at: /home/user/hadoop/
/home/user/hadoop/jar1,
/home/user/hadoop/jar2,
.
.
.
/home/user/hadoop/jarN,
/home/user/hadoop/lib/jar1,
/home/user/hadoop/lib/jar2,
/home/user/hadoop/lib/jarN,
1./home/user/app/www/WEB-INF/classes,
2/${mapred.local.dir}/taskTracker/{user}/jobcache/{jobid}/jars/Myjar.jar
[note:- basically this has the modified class that I need to use for my
Map-Reduce job]
Well its clear from this classpath that i will end up using the
Application.class from the classes folder. with gives me incorrect results.
Now my Question is, how do I make sure i reverse the order of 1 & 2.
Some pointer that I found was,
1) if MyJar.jar is not changing much then I can put in a shared location and
modify my hadoop-env.sh to
export
HADOOP_CLASSPATH="/some/share/location/lib:/home/user/app/www/WEB-INF/classes:$HADOOP_CLASSPATH"
2) get rid of /home/user/app/www/WEB-INF/classes, from my hadoop-env.sh
3) is there any property taht suggest to add before classpath ?
Any help is greatly appreciated.
To Summarize,
If I have HADOOP_CLASSPTH in hadoop-env.sh already set, then how do I add
application jar before this classpath.
Again. I saw the DistributedCache.java [hadoop src] and the code looks like.
public static void addFileToClassPath(Path file, Configuration conf)
throws IOException {
String classpath = conf.get("mapred.job.classpath.files");
conf.set("mapred.job.classpath.files", classpath == null ? file
.toString() : classpath + System.getProperty("path.separator")
+ file.toString());
.............
}
basically new files are added to the end of existing classpath.
Thanks,
Dharin.
--
View this message in context: http://old.nabble.com/Hadoopp_ClassPath-issue.-tp33666009p33666009.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Hadoopp_ClassPath issue.
Posted by John George <jo...@yahoo-inc.com>.
Dharin,
I believe the properties you are looking for are the following:
HADOOP_USER_CLASSPATH_FIRST: When defined, this will let the user
suggested classpath to the beginning of global classpath. So, you would
have to do something like 'export HADOOP_USER_CLASSPATH_FIRST=true'. If
you are on 2.0 (or 0.23), please refer bin/hadoop-config.sh for more
information. If you are on 1.0 (or 0.20), refer to hadoop script.
Now, if you want to run an M/R job by passing your own jar and you want
that jar to be used first, you want to set the config parameter
'mapreduce.job.user.classpath.first' and then the user provided jar will
be put in before $HADOOP_CLASSPATH.
Hope this makes sense.
Also, these will work on 1.0 (or 0.23) above.
Refer:
https://issues.apache.org/jira/browse/MAPREDUCE-3696 (for 2.0, 0.23)
https://issues.apache.org/jira/browse/MAPREDUCE-1938 (1.0, 0.20)
Thanks,
John George
-----Original Message-----
From: dmaniar <dh...@gmail.com>
Reply-To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>
Date: Tue, 10 Apr 2012 21:09:10 -0700
To: "core-user@hadoop.apache.org" <co...@hadoop.apache.org>
Subject: Hadoopp_ClassPath issue.
>
>Hi,
>
>I am new to hadoop and its not very familiar with internal working. I had
>some questions about HADOOP_CLASSPATH.
>
>We are currently suppose to use a Hadoop cluster with 4 machines and its
>HADOOP_CLASSPATH in hadoop-env.sh is as below.
>export
>HADOOP_CLASSPATH="/home/user/app/www/WEB-INF/classes:$HADOOP_CLASSPATH"
>
>Now my,
>/home/user/app/www/WEB-INF/classes has a class called Application.class
>
>From a remote machine I submit a map-reduce job to this cluster, with a
>jar
>called MyJar.jar. [This has a Application.class too, but with some
>modifications]
>
>When the TaskTracker spawns a child Java process for the Mapper the
>classpath I see is as below in that order,
>
>Lets say my hadoop is installed at: /home/user/hadoop/
>/home/user/hadoop/jar1,
>/home/user/hadoop/jar2,
>.
>.
>.
>/home/user/hadoop/jarN,
>/home/user/hadoop/lib/jar1,
>/home/user/hadoop/lib/jar2,
>/home/user/hadoop/lib/jarN,
>1./home/user/app/www/WEB-INF/classes,
>2/${mapred.local.dir}/taskTracker/{user}/jobcache/{jobid}/jars/Myjar.jar
>[note:- basically this has the modified class that I need to use for my
>Map-Reduce job]
>
>Well its clear from this classpath that i will end up using the
>Application.class from the classes folder. with gives me incorrect
>results.
>
>Now my Question is, how do I make sure i reverse the order of 1 & 2.
>
>Some pointer that I found was,
>1) if MyJar.jar is not changing much then I can put in a shared location
>and
>modify my hadoop-env.sh to
>export
>HADOOP_CLASSPATH="/some/share/location/lib:/home/user/app/www/WEB-INF/clas
>ses:$HADOOP_CLASSPATH"
>
>2) get rid of /home/user/app/www/WEB-INF/classes, from my hadoop-env.sh
>
>3) is there any property taht suggest to add before classpath ?
>
>Any help is greatly appreciated.
>
>To Summarize,
>If I have HADOOP_CLASSPTH in hadoop-env.sh already set, then how do I add
>application jar before this classpath.
>
>Again. I saw the DistributedCache.java [hadoop src] and the code looks
>like.
>
>public static void addFileToClassPath(Path file, Configuration conf)
> throws IOException {
> String classpath = conf.get("mapred.job.classpath.files");
> conf.set("mapred.job.classpath.files", classpath == null ? file
> .toString() : classpath + System.getProperty("path.separator")
> + file.toString());
> .............
>}
>
>basically new files are added to the end of existing classpath.
>
>
>Thanks,
>Dharin.
>
>
>
>
>--
>View this message in context:
>http://old.nabble.com/Hadoopp_ClassPath-issue.-tp33666009p33666009.html
>Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
Re: Hadoopp_ClassPath issue.
Posted by John George <jo...@yahoo-inc.com>.
Dharin,
I believe the properties you are looking for are the following:
HADOOP_USER_CLASSPATH_FIRST: When defined, this will let the user
suggested classpath to the beginning of global classpath. So, you would
have to do something like 'export HADOOP_USER_CLASSPATH_FIRST=true'. If
you are on 2.0 (or 0.23), please refer bin/hadoop-config.sh for more
information. If you are on 1.0 (or 0.20), refer to hadoop script.
Now, if you want to run an M/R job by passing your own jar and you want
that jar to be used first, you want to set the config parameter
'mapreduce.job.user.classpath.first' and then the user provided jar will
be put in before $HADOOP_CLASSPATH.
Hope this makes sense.
Also, these will work on 1.0 (or 0.23) above.
Refer:
https://issues.apache.org/jira/browse/MAPREDUCE-3696 (for 2.0, 0.23)
https://issues.apache.org/jira/browse/MAPREDUCE-1938 (1.0, 0.20)
Thanks,
John George
-----Original Message-----
From: dmaniar <dh...@gmail.com>
Reply-To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>
Date: Tue, 10 Apr 2012 21:09:10 -0700
To: "core-user@hadoop.apache.org" <co...@hadoop.apache.org>
Subject: Hadoopp_ClassPath issue.
>
>Hi,
>
>I am new to hadoop and its not very familiar with internal working. I had
>some questions about HADOOP_CLASSPATH.
>
>We are currently suppose to use a Hadoop cluster with 4 machines and its
>HADOOP_CLASSPATH in hadoop-env.sh is as below.
>export
>HADOOP_CLASSPATH="/home/user/app/www/WEB-INF/classes:$HADOOP_CLASSPATH"
>
>Now my,
>/home/user/app/www/WEB-INF/classes has a class called Application.class
>
>From a remote machine I submit a map-reduce job to this cluster, with a
>jar
>called MyJar.jar. [This has a Application.class too, but with some
>modifications]
>
>When the TaskTracker spawns a child Java process for the Mapper the
>classpath I see is as below in that order,
>
>Lets say my hadoop is installed at: /home/user/hadoop/
>/home/user/hadoop/jar1,
>/home/user/hadoop/jar2,
>.
>.
>.
>/home/user/hadoop/jarN,
>/home/user/hadoop/lib/jar1,
>/home/user/hadoop/lib/jar2,
>/home/user/hadoop/lib/jarN,
>1./home/user/app/www/WEB-INF/classes,
>2/${mapred.local.dir}/taskTracker/{user}/jobcache/{jobid}/jars/Myjar.jar
>[note:- basically this has the modified class that I need to use for my
>Map-Reduce job]
>
>Well its clear from this classpath that i will end up using the
>Application.class from the classes folder. with gives me incorrect
>results.
>
>Now my Question is, how do I make sure i reverse the order of 1 & 2.
>
>Some pointer that I found was,
>1) if MyJar.jar is not changing much then I can put in a shared location
>and
>modify my hadoop-env.sh to
>export
>HADOOP_CLASSPATH="/some/share/location/lib:/home/user/app/www/WEB-INF/clas
>ses:$HADOOP_CLASSPATH"
>
>2) get rid of /home/user/app/www/WEB-INF/classes, from my hadoop-env.sh
>
>3) is there any property taht suggest to add before classpath ?
>
>Any help is greatly appreciated.
>
>To Summarize,
>If I have HADOOP_CLASSPTH in hadoop-env.sh already set, then how do I add
>application jar before this classpath.
>
>Again. I saw the DistributedCache.java [hadoop src] and the code looks
>like.
>
>public static void addFileToClassPath(Path file, Configuration conf)
> throws IOException {
> String classpath = conf.get("mapred.job.classpath.files");
> conf.set("mapred.job.classpath.files", classpath == null ? file
> .toString() : classpath + System.getProperty("path.separator")
> + file.toString());
> .............
>}
>
>basically new files are added to the end of existing classpath.
>
>
>Thanks,
>Dharin.
>
>
>
>
>--
>View this message in context:
>http://old.nabble.com/Hadoopp_ClassPath-issue.-tp33666009p33666009.html
>Sent from the Hadoop core-user mailing list archive at Nabble.com.
>