You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Michael Basnight <mb...@gmail.com> on 2010/02/03 22:09:06 UTC
Maven and Mini MR Cluster
Im using maven to run all my unit tests, and i have a unit test that creates a mini mr cluster. When i create this cluster, i get classdefnotfound errors for the core hadoop libs (Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Child). When i run the same test w/o creating the mini cluster, well.. it works fine. My HADOOP_HOME is set to the same version as my mvn repo, and points to a valid installation of hadoop. When i validate the classpath thru maven, (dependency:build-classpath), it says that the core libs are on the classpath as well (sourced from my .m2 repository). I just cant figure out why hadoop's mini cluster cant find those jars. Running hadoop 0.20.0.
Any suggestions?
Re: Maven and Mini MR Cluster
Posted by Michael Basnight <mb...@gmail.com>.
Ya with the hadoop_home stuff i was grasping at straws. My mini MR Cluster has a valid classpath i assume, since my entire test runs (thru 3 mapreduce jobs via the localrunner) before it gets to the mini MR cluster portion. Is it possible to print out the classpath thru the JVMManager or anything else like that for debugging purposes?
mb
On Feb 4, 2010, at 5:55 AM, Steve Loughran wrote:
> Michael Basnight wrote:
>> Im using maven to run all my unit tests, and i have a unit test that creates a mini mr cluster. When i create this cluster, i get classdefnotfound errors for the core hadoop libs (Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Child). When i run the same test w/o creating the mini cluster, well.. it works fine. My HADOOP_HOME is set to the same version as my mvn repo, and points to a valid installation of hadoop. When i validate the classpath thru maven, (dependency:build-classpath), it says that the core libs are on the classpath as well (sourced from my .m2 repository). I just cant figure out why hadoop's mini cluster cant find those jars. Running hadoop 0.20.0. Any suggestions?
>
> the miniMR cluster does everything in memory, and doesnt look at HADOOP_HOME, which is only for the shell scripts.
>
> It sounds like you need hadoop-mapreduce on your classpath. Sounds like. the Child class is the entry point used when creating new JVMs, and it is that classpath that isn't right, which is a forked JVM from the one the MiniMRCluster was created in.
Re: Maven and Mini MR Cluster
Posted by Edward Capriolo <ed...@gmail.com>.
On Thu, Feb 4, 2010 at 12:12 PM, Steve Loughran <st...@apache.org> wrote:
> Michael Basnight wrote:
>>
>> Ya with the hadoop_home stuff i was grasping at straws. My mini MR Cluster
>> has a valid classpath i assume, since my entire test runs (thru 3 mapreduce
>> jobs via the localrunner) before it gets to the mini MR cluster portion. Is
>> it possible to print out the classpath thru the JVMManager or anything else
>> like that for debugging purposes?
>>
>
> probably, though I don't know what.
>
>
Normally from a shell script, I do something like this to ensure I
suck up hadoop.jar, hadoop-test.jar, and its dependents.
CPATH=""
for f in /opt/hadoop/lib/*.jar ; do
CPATH="${CPATH}:$f"
done
for f in /opt/hadoop/*.jar ; do
CPATH="${CPATH}:$f"
done
java -cp $CPATH
If you are in the build phase you should refer to the build.xml and
build-common and try to emulate that classpath and add you needs.
Re: Maven and Mini MR Cluster
Posted by Steve Loughran <st...@apache.org>.
Michael Basnight wrote:
> Ya with the hadoop_home stuff i was grasping at straws. My mini MR Cluster has a valid classpath i assume, since my entire test runs (thru 3 mapreduce jobs via the localrunner) before it gets to the mini MR cluster portion. Is it possible to print out the classpath thru the JVMManager or anything else like that for debugging purposes?
>
probably, though I don't know what.
Re: Maven and Mini MR Cluster
Posted by Michael Basnight <mb...@gmail.com>.
Ya with the hadoop_home stuff i was grasping at straws. My mini MR Cluster has a valid classpath i assume, since my entire test runs (thru 3 mapreduce jobs via the localrunner) before it gets to the mini MR cluster portion. Is it possible to print out the classpath thru the JVMManager or anything else like that for debugging purposes?
mb
On Feb 4, 2010, at 5:55 AM, Steve Loughran wrote:
> Michael Basnight wrote:
>> Im using maven to run all my unit tests, and i have a unit test that creates a mini mr cluster. When i create this cluster, i get classdefnotfound errors for the core hadoop libs (Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Child). When i run the same test w/o creating the mini cluster, well.. it works fine. My HADOOP_HOME is set to the same version as my mvn repo, and points to a valid installation of hadoop. When i validate the classpath thru maven, (dependency:build-classpath), it says that the core libs are on the classpath as well (sourced from my .m2 repository). I just cant figure out why hadoop's mini cluster cant find those jars. Running hadoop 0.20.0. Any suggestions?
>
> the miniMR cluster does everything in memory, and doesnt look at HADOOP_HOME, which is only for the shell scripts.
>
> It sounds like you need hadoop-mapreduce on your classpath. Sounds like. the Child class is the entry point used when creating new JVMs, and it is that classpath that isn't right, which is a forked JVM from the one the MiniMRCluster was created in.
Re: Maven and Mini MR Cluster
Posted by Steve Loughran <st...@apache.org>.
Michael Basnight wrote:
> Im using maven to run all my unit tests, and i have a unit test that creates a mini mr cluster. When i create this cluster, i get classdefnotfound errors for the core hadoop libs (Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Child). When i run the same test w/o creating the mini cluster, well.. it works fine. My HADOOP_HOME is set to the same version as my mvn repo, and points to a valid installation of hadoop. When i validate the classpath thru maven, (dependency:build-classpath), it says that the core libs are on the classpath as well (sourced from my .m2 repository). I just cant figure out why hadoop's mini cluster cant find those jars. Running hadoop 0.20.0.
>
> Any suggestions?
the miniMR cluster does everything in memory, and doesnt look at
HADOOP_HOME, which is only for the shell scripts.
It sounds like you need hadoop-mapreduce on your classpath. Sounds like.
the Child class is the entry point used when creating new JVMs, and it
is that classpath that isn't right, which is a forked JVM from the one
the MiniMRCluster was created in.