You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Praveen Sripati <pr...@gmail.com> on 2011/08/10 04:39:34 UTC
MRv2 Configuration in Eclipse
Hi,
Are there any instructions to start the ResourceManager, NodeManager,
HistoryServer from Eclipse? I got the code from SVN, compiled it and ran the
sample program. The projects have also been exported in Eclipse. I want to
run the RM, NM and HS from Eclipse, so as to see the flow and fix any
problems in the code.
Thanks,
Praveen
Re: MRv2 Configuration in Eclipse
Posted by Praveen Sripati <pr...@gmail.com>.
Hi,
I did spend some time on trying to run RM, NM and HS from within Eclipse,
but got partial success. Here are the steps I followed (just in case someone
is interested)
- Got the code from SVN.
- Ran the ant and mvn command for compiling Hadoop Common, HDFS and
MapReduce.
- Ran the 'mvn eclipse:eclipse' and 'ant eclipse' for the projects.
- Included the following in /etc/environment (a reboot was required for it
to take effect)
HADOOP_MAPRED_HOME="/home/praveensripati/Hadoop/trunk/mapreduce"
HADOOP_COMMON_HOME="/home/praveensripati/Hadoop/trunk/hadoop-common"
HADOOP_HDFS_HOME="/home/praveensripati/Hadoop/trunk/hdfs"
YARN_HOME="/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT"
HADOOP_CONF_DIR="/home/praveensripati/Hadoop/conf"
YARN_CONF_DIR="/home/praveensripati/Hadoop/conf"
YARN_INSTALL="/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT"
JAVA_HOME="/usr/lib/jvm/java-6-sun"
- Imported the java projects in Eclipse.
- In the 'Java Build Path' removed the jar dependencies in the Libraries Tab
and included the corresponding projects in the Sources Tab. Made sure that
the projects got compiled properly.
- Started the resourcemanager, nodemanager and historyserver from command
prompt using the yarn-daemon.sh command. This dumps out the java command
with the arguments. Stopped the resourcemanager, nodemanager and
historyserver again from the command propmpt.
- Ran the ResourceManager, NodeManager and JobHistoryServer from Eclipse,
passing the parameters got from the previous step.
- Created a Java project with Eclipse and copied
org.apache.hadoop.examples.RandomWriter.java and ran with the following
parameters
-Dmapreduce.job.user.name=praveensripati
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912
/home/praveensripati/Hadoop/trunk/hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT-bin/bin/output
and got the following exceptions for each of the mapper invocation. Could
not figure out the cause of this exception.
2011-08-13 21:44:35,785 INFO mapreduce.Job (Job.java:printTaskEvents(1213))
- Task Id : attempt_1313250752016_0003_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:180)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
... 8 more
- But was able to run the RandomWriter from command propmpt using the below
command
./hadoop jar
$HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
$YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output
Thanks,
Praveen
On Sat, Aug 13, 2011 at 10:40 PM, Praveen Sripati
<pr...@gmail.com>wrote:
>
> Hi,
>
> I am able to run the RM, NM and HS from Eclipse with a couple of settings
> and run the org.apache.hadoop.examples.RandomWriter from command promptusing the 'hadoop jar' command. Would be nice for easier development and
> debugging of these components.
>
> But, when I run the org.apache.hadoop.examples.RandomWriter from Eclipse, I
> get the below exceptions. I created a new Java project in Eclipse and copied
> the RandomWriter.java to it. This exception is getting repeated map number
> of times, looks like the error is thrown for every map launch.
>
> Not sure what this error means, any help is appreciated.
>
> 2011-08-13 21:44:35,785 INFO mapreduce.Job
> (Job.java:printTaskEvents(1213)) - Task Id :
> attempt_1313250752016_0003_m_000001_2, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.RandomWriter$RandomMapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:180)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.RandomWriter$RandomMapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
> ... 8 more
>
> Thanks,
> Praveen
>
> On Wed, Aug 10, 2011 at 10:56 AM, Arun C Murthy <ac...@hortonworks.com>wrote:
>
>> Praveen,
>>
>> I'm not aware of any devs using eclipse to run MRv2, I haven't tried it
>> myself.
>>
>> OTOH, if you do get to we are happy to follow your lead, please feel free
>> to provide us instructions and we can update the README... thanks.
>>
>> Arun
>>
>> On Aug 9, 2011, at 7:39 PM, Praveen Sripati wrote:
>>
>> > Hi,
>> >
>> > Are there any instructions to start the ResourceManager, NodeManager,
>> > HistoryServer from Eclipse? I got the code from SVN, compiled it and ran
>> the
>> > sample program. The projects have also been exported in Eclipse. I want
>> to
>> > run the RM, NM and HS from Eclipse, so as to see the flow and fix any
>> > problems in the code.
>> >
>> > Thanks,
>> > Praveen
>>
>>
>
Re: MRv2 Configuration in Eclipse
Posted by Praveen Sripati <pr...@gmail.com>.
Hi,
I am able to run the RM, NM and HS from Eclipse with a couple of settings
and run the org.apache.hadoop.examples.RandomWriter from command
promptusing the 'hadoop jar' command. Would be nice for easier
development and
debugging of these components.
But, when I run the org.apache.hadoop.examples.RandomWriter from Eclipse, I
get the below exceptions. I created a new Java project in Eclipse and copied
the RandomWriter.java to it. This exception is getting repeated map number
of times, looks like the error is thrown for every map launch.
Not sure what this error means, any help is appreciated.
2011-08-13 21:44:35,785 INFO mapreduce.Job (Job.java:printTaskEvents(1213))
- Task Id : attempt_1313250752016_0003_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:180)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
... 8 more
Thanks,
Praveen
On Wed, Aug 10, 2011 at 10:56 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Praveen,
>
> I'm not aware of any devs using eclipse to run MRv2, I haven't tried it
> myself.
>
> OTOH, if you do get to we are happy to follow your lead, please feel free
> to provide us instructions and we can update the README... thanks.
>
> Arun
>
> On Aug 9, 2011, at 7:39 PM, Praveen Sripati wrote:
>
> > Hi,
> >
> > Are there any instructions to start the ResourceManager, NodeManager,
> > HistoryServer from Eclipse? I got the code from SVN, compiled it and ran
> the
> > sample program. The projects have also been exported in Eclipse. I want
> to
> > run the RM, NM and HS from Eclipse, so as to see the flow and fix any
> > problems in the code.
> >
> > Thanks,
> > Praveen
>
>
Re: MRv2 Configuration in Eclipse
Posted by Praveen Sripati <pr...@gmail.com>.
Arun,
I did try it out for sometime, but didn't work. Need to get more into the
code to fix the problem.
It's real difficult to use some editor instead of an IDE to do the
development, bug fixing or get familiar with the code.
I was able to run the original MapReduce applications from within Eclipse.
I created a JIRA for the same.
https://issues.apache.org/jira/browse/MAPREDUCE-2798
Thanks,
Praveen
On Wed, Aug 10, 2011 at 10:56 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Praveen,
>
> I'm not aware of any devs using eclipse to run MRv2, I haven't tried it
> myself.
>
> OTOH, if you do get to we are happy to follow your lead, please feel free
> to provide us instructions and we can update the README... thanks.
>
> Arun
>
> On Aug 9, 2011, at 7:39 PM, Praveen Sripati wrote:
>
> > Hi,
> >
> > Are there any instructions to start the ResourceManager, NodeManager,
> > HistoryServer from Eclipse? I got the code from SVN, compiled it and ran
> the
> > sample program. The projects have also been exported in Eclipse. I want
> to
> > run the RM, NM and HS from Eclipse, so as to see the flow and fix any
> > problems in the code.
> >
> > Thanks,
> > Praveen
>
>
Re: MRv2 Configuration in Eclipse
Posted by Arun C Murthy <ac...@hortonworks.com>.
Praveen,
I'm not aware of any devs using eclipse to run MRv2, I haven't tried it myself.
OTOH, if you do get to we are happy to follow your lead, please feel free to provide us instructions and we can update the README... thanks.
Arun
On Aug 9, 2011, at 7:39 PM, Praveen Sripati wrote:
> Hi,
>
> Are there any instructions to start the ResourceManager, NodeManager,
> HistoryServer from Eclipse? I got the code from SVN, compiled it and ran the
> sample program. The projects have also been exported in Eclipse. I want to
> run the RM, NM and HS from Eclipse, so as to see the flow and fix any
> problems in the code.
>
> Thanks,
> Praveen