You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Praveen Sripati <pr...@gmail.com> on 2011/12/21 18:34:51 UTC
ResourceManager not getting started in 0.23
Hi,
I am making some change to the 0.23 branch and got the latest code and
successfully build it. Am able to start the NameNode and the DataNode, but
when I start the ResourceManager I am getting 'ClassNotFoundException'. The
classpath to run the ResourceManager is not getting generated properly by
the 'sbin/yarn-daemon.sh' command. I also noticed that the modules folder
is not getting created when the hadoop-0.23.1-SNAPSHOT.tar.gz is untarred.
Is this a bug or I missed something?
---------------------
sbin/yarn-daemon.sh start resourcemanager
-classpath
/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf//rm-config/log4j.properties
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
---------------------
BTW, I am able to run the example Hadoop after downloading
hadoop-0.23.0.tar.gz from Apache.
I have the following settings in .bashrc
export
HADOOP_DEV_HOME=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT
export HADOOP_MAPRED_HOME=$HADOOP_DEV_HOME
export HADOOP_COMMON_HOME=$HADOOP_DEV_HOME
export HADOOP_HDFS_HOME=$HADOOP_DEV_HOME
export YARN_HOME=$HADOOP_DEV_HOME
export HADOOP_CONF_DIR=$HADOOP_DEV_HOME/conf/
export YARN_CONF_DIR=~$HADOOP_DEV_HOME/conf/
Regards,
Praveen
Re: ResourceManager not getting started in 0.23
Posted by Patrick Wendell <pw...@gmail.com>.
I had the same problem. Solved by re-adding the following lines back
to bin/yarn right after CLASSPATH is first declared:
>>>>
# add hadoop-common libs to CLASSPATH
if [ ! -d "$HADOOP_COMMON_HOME" ]; then
echo No HADOOP_COMMON_HOME set.
echo Please specify it either in yarn-env.sh or in the environment.
exit 1
fi
CLASSPATH=${CLASSPATH}:$HADOOP_COMMON_HOME/share/hadoop/common'/*'
CLASSPATH=${CLASSPATH}:$HADOOP_COMMON_HOME/share/hadoop/common/lib'/*'
>>>
I've done a clean and re-build and I still need to explicitly include
those hadoop common JARS or else the RM won't start. Not sure what is
actually the problem here.
- Patrick
On Wed, Dec 21, 2011 at 8:25 PM, Praveen Sripati
<pr...@gmail.com> wrote:
> Arun,
>
> Thanks for the response.
>
> I had to explicitly add the hadoop-common-0.23.1-SNAPSHOT.jar in the
> classpath using the below commands.
>
> cd $HADOOP_COMMON_HOME/share/hadoop/mapreduce/
> ln -s ../common/hadoop-common-0.23.1-SNAPSHOT.jar
>
> But, now ResourceManager fails silently with the following output in the
> yarn-praveensripati-resourcemanager-praveen-laptop.out. When I run jps, I
> don't see ResourceManager.
>
> /usr/lib/jvm/jdk1.6.0_27/bin/java -Dproc_resourcemanager -Xmx1000m
> -Dhadoop.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> -Dyarn.logs.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> -Dhadoop.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> -Dyarn.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> -Dyarn.home.dir= -Dyarn.id.str=praveensripati
> -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Dyarn.policy.file=hadoop-policy.xml
> -Dhadoop.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> -Dyarn.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> -Dhadoop.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> -Dyarn.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> -Dyarn.home.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT
> -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -classpath
> /home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/share/hadoop/mapreduce/*:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/share/hadoop/mapreduce/lib/*:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/rm-config/log4j.properties
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.yarn.server.resourcemanager.ResourceManager).
> log4j:WARN Please initialize the log4j system properly.
>
> Regards,
> Praveen
>
> On Wed, Dec 21, 2011 at 11:08 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
>> Two ways around:
>>
>> # You might have a stale version of hadoop-common, apparently that should
>> fix it.
>> # Upgrade to include MAPREDUCE-3588 which got committed yesterday.
>>
>> Arun
>>
>>
>> On Dec 21, 2011, at 9:34 AM, Praveen Sripati wrote:
>>
>> > Hi,
>> >
>> > I am making some change to the 0.23 branch and got the latest code and
>> > successfully build it. Am able to start the NameNode and the DataNode,
>> but
>> > when I start the ResourceManager I am getting 'ClassNotFoundException'.
>> The
>> > classpath to run the ResourceManager is not getting generated properly by
>> > the 'sbin/yarn-daemon.sh' command. I also noticed that the modules folder
>> > is not getting created when the hadoop-0.23.1-SNAPSHOT.tar.gz is
>> untarred.
>> >
>> > Is this a bug or I missed something?
>> >
>> > ---------------------
>> >
>> > sbin/yarn-daemon.sh start resourcemanager
>> >
>> > -classpath
>> >
>> /home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf//rm-config/log4j.properties
>> >
>> > Caused by: java.lang.ClassNotFoundException:
>> > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> >
>> > ---------------------
>> >
>> > BTW, I am able to run the example Hadoop after downloading
>> > hadoop-0.23.0.tar.gz from Apache.
>> >
>> > I have the following settings in .bashrc
>> >
>> > export
>> >
>> HADOOP_DEV_HOME=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT
>> > export HADOOP_MAPRED_HOME=$HADOOP_DEV_HOME
>> > export HADOOP_COMMON_HOME=$HADOOP_DEV_HOME
>> > export HADOOP_HDFS_HOME=$HADOOP_DEV_HOME
>> > export YARN_HOME=$HADOOP_DEV_HOME
>> > export HADOOP_CONF_DIR=$HADOOP_DEV_HOME/conf/
>> > export YARN_CONF_DIR=~$HADOOP_DEV_HOME/conf/
>> >
>> > Regards,
>> > Praveen
>>
>>
Re: ResourceManager not getting started in 0.23
Posted by Praveen Sripati <pr...@gmail.com>.
Arun,
Thanks for the response.
I had to explicitly add the hadoop-common-0.23.1-SNAPSHOT.jar in the
classpath using the below commands.
cd $HADOOP_COMMON_HOME/share/hadoop/mapreduce/
ln -s ../common/hadoop-common-0.23.1-SNAPSHOT.jar
But, now ResourceManager fails silently with the following output in the
yarn-praveensripati-resourcemanager-praveen-laptop.out. When I run jps, I
don't see ResourceManager.
/usr/lib/jvm/jdk1.6.0_27/bin/java -Dproc_resourcemanager -Xmx1000m
-Dhadoop.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
-Dyarn.logs.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
-Dhadoop.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
-Dyarn.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
-Dyarn.home.dir= -Dyarn.id.str=praveensripati
-Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
-Dyarn.policy.file=hadoop-policy.xml
-Dhadoop.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
-Dyarn.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
-Dhadoop.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
-Dyarn.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
-Dyarn.home.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT
-Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -classpath
/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/share/hadoop/mapreduce/*:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/share/hadoop/mapreduce/lib/*:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/rm-config/log4j.properties
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.yarn.server.resourcemanager.ResourceManager).
log4j:WARN Please initialize the log4j system properly.
Regards,
Praveen
On Wed, Dec 21, 2011 at 11:08 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Two ways around:
>
> # You might have a stale version of hadoop-common, apparently that should
> fix it.
> # Upgrade to include MAPREDUCE-3588 which got committed yesterday.
>
> Arun
>
>
> On Dec 21, 2011, at 9:34 AM, Praveen Sripati wrote:
>
> > Hi,
> >
> > I am making some change to the 0.23 branch and got the latest code and
> > successfully build it. Am able to start the NameNode and the DataNode,
> but
> > when I start the ResourceManager I am getting 'ClassNotFoundException'.
> The
> > classpath to run the ResourceManager is not getting generated properly by
> > the 'sbin/yarn-daemon.sh' command. I also noticed that the modules folder
> > is not getting created when the hadoop-0.23.1-SNAPSHOT.tar.gz is
> untarred.
> >
> > Is this a bug or I missed something?
> >
> > ---------------------
> >
> > sbin/yarn-daemon.sh start resourcemanager
> >
> > -classpath
> >
> /home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf//rm-config/log4j.properties
> >
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> >
> > ---------------------
> >
> > BTW, I am able to run the example Hadoop after downloading
> > hadoop-0.23.0.tar.gz from Apache.
> >
> > I have the following settings in .bashrc
> >
> > export
> >
> HADOOP_DEV_HOME=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT
> > export HADOOP_MAPRED_HOME=$HADOOP_DEV_HOME
> > export HADOOP_COMMON_HOME=$HADOOP_DEV_HOME
> > export HADOOP_HDFS_HOME=$HADOOP_DEV_HOME
> > export YARN_HOME=$HADOOP_DEV_HOME
> > export HADOOP_CONF_DIR=$HADOOP_DEV_HOME/conf/
> > export YARN_CONF_DIR=~$HADOOP_DEV_HOME/conf/
> >
> > Regards,
> > Praveen
>
>
Re: ResourceManager not getting started in 0.23
Posted by Arun C Murthy <ac...@hortonworks.com>.
Two ways around:
# You might have a stale version of hadoop-common, apparently that should fix it.
# Upgrade to include MAPREDUCE-3588 which got committed yesterday.
Arun
On Dec 21, 2011, at 9:34 AM, Praveen Sripati wrote:
> Hi,
>
> I am making some change to the 0.23 branch and got the latest code and
> successfully build it. Am able to start the NameNode and the DataNode, but
> when I start the ResourceManager I am getting 'ClassNotFoundException'. The
> classpath to run the ResourceManager is not getting generated properly by
> the 'sbin/yarn-daemon.sh' command. I also noticed that the modules folder
> is not getting created when the hadoop-0.23.1-SNAPSHOT.tar.gz is untarred.
>
> Is this a bug or I missed something?
>
> ---------------------
>
> sbin/yarn-daemon.sh start resourcemanager
>
> -classpath
> /home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf//rm-config/log4j.properties
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
>
> ---------------------
>
> BTW, I am able to run the example Hadoop after downloading
> hadoop-0.23.0.tar.gz from Apache.
>
> I have the following settings in .bashrc
>
> export
> HADOOP_DEV_HOME=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT
> export HADOOP_MAPRED_HOME=$HADOOP_DEV_HOME
> export HADOOP_COMMON_HOME=$HADOOP_DEV_HOME
> export HADOOP_HDFS_HOME=$HADOOP_DEV_HOME
> export YARN_HOME=$HADOOP_DEV_HOME
> export HADOOP_CONF_DIR=$HADOOP_DEV_HOME/conf/
> export YARN_CONF_DIR=~$HADOOP_DEV_HOME/conf/
>
> Regards,
> Praveen