You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Praveen Sripati <pr...@gmail.com> on 2011/08/09 05:27:47 UTC

Not able to execute MRv2

Hi,

Looks like the below instructions are a bit outdated. I got the mapreduce
code from the MR-279 branch and the rest of the code from trunk. The
hadoop-mapreduce-1.0-SNAPSHOT-all.tar.gz file got generated successfully.

http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL

1) Expanded the tar file to the $YARN_INSTALL directory, created symbolic
links to the jars in the hadoop-common/lib folder and ran the
"bin/yarn-daemon.sh start resourcemanager" command from the $YARN_INSTALL
directory to get the below error

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.conf.Configuration
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class:
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.  Program will
exit.

Then copied the hadoop-common-0.23.0-SNAPSHOT.jar from the
hadoop-common/target to the hadoop-common/lib folder. Then the
yarn-praveensripati-resourcemanager-praveensripati.out had the following
error after starting the ResourceManager.

Exception in thread "main" java.lang.IllegalStateException: For this
operation, current State must be STARTED instead of INITED
        at
org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101)
        at
org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69)
        at
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher.stop(ApplicationMasterLauncher.java:90)
        at
org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
        at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:423)
        at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:493)

Started the NodeManager and the HistoryServer. Running the jps shows only
the ResourceManager and the NodeManager and not the HistoryServer.

2) From the hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT-bin/bin
folder I get the following error running the sample program

./hadoop jar
$HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
$YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output

java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/lib/partition/InputSampler$Sampler
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
    at java.lang.Class.getMethod0(Class.java:2670)
    at java.lang.Class.getMethod(Class.java:1603)
    at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.<init>(ProgramDriver.java:60)

The hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar which has the
InputSampler.java class is there in the hadoop-common/lib folder, but still
getting the above error.

None of the .log files are getting generated. So, it's getting difficult yo
debug. Appreciate help in resolving this.

Thanks,
Praveen

Re: Not able to execute MRv2

Posted by Mahadev Konar <ma...@hortonworks.com>.
Hi Praveen,
 This is an issue because of incompatible versions of avro. This occurs if you have avro 1.3.2 in your cp. We use avro 1.4.

For the time being you can remove any occurence of avro 1.3.2. jar in your classpath and that should get you out of the error below.

For fixing it  in the right manner there is a jira open:

https://issues.apache.org/jira/browse/HADOOP-7264

Fixing that should get rid of this error.

Hope that helps.

thanks
mahadev

On Aug 9, 2011, at 7:57 AM, Praveen Sripati wrote:

> Hi,
> 
> For some reason none of the .log files are getting generated, so I had to
> put some println and figure out the problems. I just see the .out files for
> the RM, NM and the HistoryServer.
> 
> 1)  I get the below error while starting the ResourceManager. In the
> hadoop-mapreduce-1.0-SNAPSHOT/lib folder, there were avro-1.3.2.jar and
> avro-1.4.1.jar files. Removing the 1.3.2 version of avro fixed the problem.
> When I run JPS, I see RM, NM and HistoryServer.
> 
> java.lang.NoSuchMethodError: org.apache.avro.ipc.Server.start()V
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.start(ResourceTrackerService.java:128)
> at
> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
> 
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> 
> 2) To run the sample program, had to create symbolic links for some of the
> jars in the
> /home/praveensripati/Hadoop/trunk/hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT-bin/share/hadoop/common
> 
> When I ran the "hadoop classpath" command, it had some of the jars in the
> above folder.
> 
> What would be the reason for the log files not being generated? I would like
> to see the .log files to see the flow of the MRv2.
> 
> Thanks,
> Praveen
> 
> On Tue, Aug 9, 2011 at 8:57 AM, Praveen Sripati <pr...@gmail.com>wrote:
> 
>> 
>> Hi,
>> 
>> Looks like the below instructions are a bit outdated. I got the mapreduce
>> code from the MR-279 branch and the rest of the code from trunk. The
>> hadoop-mapreduce-1.0-SNAPSHOT-all.tar.gz file got generated successfully.
>> 
>> 
>> http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL
>> 
>> 1) Expanded the tar file to the $YARN_INSTALL directory, created symbolic
>> links to the jars in the hadoop-common/lib folder and ran the
>> "bin/yarn-daemon.sh start resourcemanager" command from the $YARN_INSTALL
>> directory to get the below error
>> 
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/conf/Configuration
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.conf.Configuration
>>    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>    at java.security.AccessController.doPrivileged(Native Method)
>>    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> Could not find the main class:
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.  Program will
>> exit.
>> 
>> Then copied the hadoop-common-0.23.0-SNAPSHOT.jar from the
>> hadoop-common/target to the hadoop-common/lib folder. Then the
>> yarn-praveensripati-resourcemanager-praveensripati.out had the following
>> error after starting the ResourceManager.
>> 
>> Exception in thread "main" java.lang.IllegalStateException: For this
>> operation, current State must be STARTED instead of INITED
>>        at
>> org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101)
>>        at
>> org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69)
>>        at
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher.stop(ApplicationMasterLauncher.java:90)
>>        at
>> org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
>>        at
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:423)
>>        at
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:493)
>> 
>> Started the NodeManager and the HistoryServer. Running the jps shows only
>> the ResourceManager and the NodeManager and not the HistoryServer.
>> 
>> 2) From the hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT-bin/bin
>> folder I get the following error running the sample program
>> 
>> ./hadoop jar
>> $HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar
>> randomwriter -Dmapreduce.job.user.name=$USER
>> -Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
>> -Ddfs.block.size=536870912 -libjars
>> $YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
>> output
>> 
>> java.lang.NoClassDefFoundError:
>> org/apache/hadoop/mapreduce/lib/partition/InputSampler$Sampler
>>    at java.lang.Class.getDeclaredMethods0(Native Method)
>>    at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>>    at java.lang.Class.getMethod0(Class.java:2670)
>>    at java.lang.Class.getMethod(Class.java:1603)
>>    at
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.<init>(ProgramDriver.java:60)
>> 
>> The hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar which has the
>> InputSampler.java class is there in the hadoop-common/lib folder, but still
>> getting the above error.
>> 
>> None of the .log files are getting generated. So, it's getting difficult yo
>> debug. Appreciate help in resolving this.
>> 
>> Thanks,
>> Praveen
>> 
>>

Re: Not able to execute MRv2

Posted by Praveen Sripati <pr...@gmail.com>.
Hi,

For some reason none of the .log files are getting generated, so I had to
put some println and figure out the problems. I just see the .out files for
the RM, NM and the HistoryServer.

1)  I get the below error while starting the ResourceManager. In the
hadoop-mapreduce-1.0-SNAPSHOT/lib folder, there were avro-1.3.2.jar and
avro-1.4.1.jar files. Removing the 1.3.2 version of avro fixed the problem.
When I run JPS, I see RM, NM and HistoryServer.

java.lang.NoSuchMethodError: org.apache.avro.ipc.Server.start()V
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.start(ResourceTrackerService.java:128)
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)

log4j:WARN No appenders could be found for logger
(org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.

2) To run the sample program, had to create symbolic links for some of the
jars in the
/home/praveensripati/Hadoop/trunk/hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT-bin/share/hadoop/common

When I ran the "hadoop classpath" command, it had some of the jars in the
above folder.

What would be the reason for the log files not being generated? I would like
to see the .log files to see the flow of the MRv2.

Thanks,
Praveen

On Tue, Aug 9, 2011 at 8:57 AM, Praveen Sripati <pr...@gmail.com>wrote:

>
> Hi,
>
> Looks like the below instructions are a bit outdated. I got the mapreduce
> code from the MR-279 branch and the rest of the code from trunk. The
> hadoop-mapreduce-1.0-SNAPSHOT-all.tar.gz file got generated successfully.
>
>
> http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL
>
> 1) Expanded the tar file to the $YARN_INSTALL directory, created symbolic
> links to the jars in the hadoop-common/lib folder and ran the
> "bin/yarn-daemon.sh start resourcemanager" command from the $YARN_INSTALL
> directory to get the below error
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/conf/Configuration
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.conf.Configuration
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.  Program will
> exit.
>
> Then copied the hadoop-common-0.23.0-SNAPSHOT.jar from the
> hadoop-common/target to the hadoop-common/lib folder. Then the
> yarn-praveensripati-resourcemanager-praveensripati.out had the following
> error after starting the ResourceManager.
>
> Exception in thread "main" java.lang.IllegalStateException: For this
> operation, current State must be STARTED instead of INITED
>         at
> org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101)
>         at
> org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69)
>         at
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher.stop(ApplicationMasterLauncher.java:90)
>         at
> org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
>         at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:423)
>         at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:493)
>
> Started the NodeManager and the HistoryServer. Running the jps shows only
> the ResourceManager and the NodeManager and not the HistoryServer.
>
> 2) From the hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT-bin/bin
> folder I get the following error running the sample program
>
> ./hadoop jar
> $HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar
> randomwriter -Dmapreduce.job.user.name=$USER
> -Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
> -Ddfs.block.size=536870912 -libjars
> $YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
> output
>
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/lib/partition/InputSampler$Sampler
>     at java.lang.Class.getDeclaredMethods0(Native Method)
>     at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>     at java.lang.Class.getMethod0(Class.java:2670)
>     at java.lang.Class.getMethod(Class.java:1603)
>     at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.<init>(ProgramDriver.java:60)
>
> The hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar which has the
> InputSampler.java class is there in the hadoop-common/lib folder, but still
> getting the above error.
>
> None of the .log files are getting generated. So, it's getting difficult yo
> debug. Appreciate help in resolving this.
>
> Thanks,
> Praveen
>
>