You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by akhil1988 <ak...@gmail.com> on 2009/07/17 21:57:11 UTC
Why /tmp directory?
Hi All,
I want to know why do we generally use tmp directory(and not any other) for
storing hdfs data, knowing the fact that tmp directory is used for storing
only temporary data?
I was wondering this because when I run a Hbase job on large data, I get
this DiskErrorExcpetion:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for
taskTracker/jobcache/job_200904072051_0001/attempt_200904072051_0001_m_000000_1/output/file.out
Please, excuse if this is a naive question.
--Akhil
--
View this message in context: http://www.nabble.com/Why--tmp-directory--tp24540694p24540694.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Why /tmp directory?
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Akhil,
That's the default configuration, but it's not meant for actual use in a
cluster. You should be manually setting dfs.data.dir, dfs.name.dir, and
mapred.local.dir on your cluster to point to the disks you want Hadoop to
use. The use of /tmp as a default is because it's a convenient location
pretty well guaranteed to be writable on most platforms, so it's handy for
just getting started with minimal configuration.
-Todd
On Fri, Jul 17, 2009 at 12:57 PM, akhil1988 <ak...@gmail.com> wrote:
>
> Hi All,
>
> I want to know why do we generally use tmp directory(and not any other) for
> storing hdfs data, knowing the fact that tmp directory is used for storing
> only temporary data?
>
> I was wondering this because when I run a Hbase job on large data, I get
> this DiskErrorExcpetion:
>
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for
>
>
> taskTracker/jobcache/job_200904072051_0001/attempt_200904072051_0001_m_000000_1/output/file.out
>
> Please, excuse if this is a naive question.
>
> --Akhil
>
> --
> View this message in context:
> http://www.nabble.com/Why--tmp-directory--tp24540694p24540694.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
Re: Hadoop logging
Posted by Aaron Kimball <aa...@cloudera.com>.
Hm. What version of Hadoop are you running? Have you modified the
log4j.properties file in other ways? The logfiles generated by Hadoop
should, by default, switch to a new file every day, appending the
previous day's date to the closed log file (e.g.,
"hadoop-hadoop-datanode-jargon.log.2009-07-13"). You should be able to
use logrotate + cron to move those files out to someplace with more
space and/or remove them after they're too old.
After you changed log4j.properties, did you restart the daemons?
Changing the properties file has no effect on running processes.
Similarly, if you're running on a proper cluster (as opposed to
standalone / pseudo-distributed modes), you'll need to set
"hadoop.root.logger=ERROR,console" in the log4j.properties file on
each separate machine; the client machine alone is not sufficient.
- Aaron
On Tue, Jul 21, 2009 at 1:59 PM, Arv Mistry<ar...@kindsight.net> wrote:
> Hi,
>
> Recently we ran out of disk space on the hadoop machine, and on
> investigation we found it was the hadoop log4j logs.
>
> In the log4j.properties file I have set the hadoop.root.logger=ERROR
> yet I still the daily hadoop-hadoopadmin-*.log files with INFO level
> logging in them. These never seem to get trimmed or rollover.
>
> Does anyone know how to limit ALL hadoop logs?
>
> Do I have to set each daemon logging individually? i.e.
>
> log4j.logger.org.apache.hadoop.mapred.JobTracker=ERROR etc
>
> Cheers Arv
>
>
>
Re: Any help? Problem in building hadoop
Posted by Gopal Gandhi <go...@yahoo.com>.
Would a guru give me a hint? I suppose this is a very common problem because this is just a build on original source code downloaded from apache.
Thanks!
________________________________
From: Gopal Gandhi <go...@yahoo.com>
To: core-dev@hadoop.apache.org
Sent: Tuesday, July 28, 2009 11:30:08 AM
Subject: Any help? Problem in building hadoop
I downloaded 0.17 source code and tried to build it but failed.
I didn't change anything in the code. Do you happen to have a clue? Thanks lot.
The error is like:
$ cd hadoop-0.17
$ ant tar
BUILD FAILED
java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.tools.ant.taskdefs.Definer.addDefinition(Definer.java:583)
at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:228)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:357)
at org.apache.tools.ant.helper.ProjectHelper2.parse(ProjectHelper2.java:142)
at org.apache.tools.ant.ProjectHelper.configureProject(ProjectHelper.java:93)
at org.apache.tools.ant.Main.runBuild(Main.java:743)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
Caused by: org.apache.commons.logging.LogConfigurationException: java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:538)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
at org.apache.jasper.JspC.<clinit>(JspC.java:90)
... 17 more
Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.apache.tools.ant.AntClassLoader.findBaseClass(AntClassLoader.java:1425)
at org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1084)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:178)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:317)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:120)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:629)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:612)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:509)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:415)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:441)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:468)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:122)
at org.apache.log4j.Logger.getLogger(Logger.java:104)
at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
... 21 more
Total time: 1 second
-G
Any help? Problem in building hadoop
Posted by Gopal Gandhi <go...@yahoo.com>.
I downloaded 0.17 source code and tried to build it but failed.
I didn't change anything in the code. Do you happen to have a clue? Thanks lot.
The error is like:
$ cd hadoop-0.17
$ ant tar
BUILD FAILED
java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.tools.ant.taskdefs.Definer.addDefinition(Definer.java:583)
at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:228)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:357)
at org.apache.tools.ant.helper.ProjectHelper2.parse(ProjectHelper2.java:142)
at org.apache.tools.ant.ProjectHelper.configureProject(ProjectHelper.java:93)
at org.apache.tools.ant.Main.runBuild(Main.java:743)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
Caused by: org.apache.commons.logging.LogConfigurationException: java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:538)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
at org.apache.jasper.JspC.<clinit>(JspC.java:90)
... 17 more
Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.apache.tools.ant.AntClassLoader.findBaseClass(AntClassLoader.java:1425)
at org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1084)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:178)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:317)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:120)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:629)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:612)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:509)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:415)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:441)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:468)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:122)
at org.apache.log4j.Logger.getLogger(Logger.java:104)
at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
... 21 more
Total time: 1 second
-G
Re: Problem in building hadoop source code
Posted by Gopal Gandhi <go...@yahoo.com>.
Would a guru give me a hint?
I suppose this is a very common problem
because this is just a build on original source code downloaded from
apache.
Thanks.
________________________________
From: Gopal Gandhi <go...@yahoo.com>
To: core-user@hadoop.apache.org
Cc: common-user@hadoop.apache.org
Sent: Tuesday, July 28, 2009 10:56:07 AM
Subject: Problem in building hadoop source code
I downloaded 0.17 source code and tried to build it but failed. The error is like:
$ cd hadoop-0.17
$ ant tar
BUILD FAILED
java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.tools.ant.taskdefs.Definer.addDefinition(Definer.java:583)
at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:228)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:357)
at org.apache.tools.ant.helper.ProjectHelper2.parse(ProjectHelper2.java:142)
at org.apache.tools.ant.ProjectHelper.configureProject(ProjectHelper.java:93)
at org.apache.tools.ant.Main.runBuild(Main.java:743)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
Caused by: org.apache.commons.logging.LogConfigurationException: java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:538)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
at org.apache.jasper.JspC.<clinit>(JspC.java:90)
... 17 more
Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.apache.tools.ant.AntClassLoader.findBaseClass(AntClassLoader.java:1425)
at org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1084)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:178)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:317)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:120)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:629)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:612)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:509)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:415)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:441)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:468)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:122)
at org.apache.log4j.Logger.getLogger(Logger.java:104)
at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
... 21 more
Total time: 1 second
I didn't change anything in the code. Do you happen to have a clue? Thanks lot.
-G
Problem in building hadoop source code
Posted by Gopal Gandhi <go...@yahoo.com>.
I downloaded 0.17 source code and tried to build it but failed. The error is like:
$ cd hadoop-0.17
$ ant tar
BUILD FAILED
java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.tools.ant.taskdefs.Definer.addDefinition(Definer.java:583)
at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:228)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:357)
at org.apache.tools.ant.helper.ProjectHelper2.parse(ProjectHelper2.java:142)
at org.apache.tools.ant.ProjectHelper.configureProject(ProjectHelper.java:93)
at org.apache.tools.ant.Main.runBuild(Main.java:743)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
Caused by: org.apache.commons.logging.LogConfigurationException: java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:538)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
at org.apache.jasper.JspC.<clinit>(JspC.java:90)
... 17 more
Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/AppenderSkeleton
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.apache.tools.ant.AntClassLoader.findBaseClass(AntClassLoader.java:1425)
at org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1084)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:178)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:317)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:120)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:629)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:612)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:509)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:415)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:441)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:468)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:122)
at org.apache.log4j.Logger.getLogger(Logger.java:104)
at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
... 21 more
Total time: 1 second
I didn't change anything in the code. Do you happen to have a clue? Thanks lot.
-G
Re: Apply a patch to hadoop
Posted by Mac Yuki <mc...@gmail.com>.
I had the same question earlier and the answer was buried in another thread.
Here's what to do:
1. Get the patch file. Place it in the base directory of your Hadoop
install (the one that has the directories src/, bin/, etc)
2. From the base directory of your Hadoop install, type "patch -p0 <
name_of_patch_file".
3. Repeat 1 and 2 if you want to apply more patches. Careful that the order
in which you apply them might matter.
4. Type "ant tar" from the base directory.
5. You will now have a directory build/ and inside there will be a file
named "hadoop-XXX.tar.gz". If you installed Hadoop by downloading a tar
ball, treat this new tar ball as you did the one you originally downloaded
to install it.
Hope that helps.
Mac
On Mon, Jul 27, 2009 at 2:50 PM, Bill Graham <bi...@gmail.com> wrote:
> http://wiki.apache.org/hadoop/HowToContribute
>
> Search for "Applying a patch" and you'll find this:
>
> patch -p0 < cool_patch.patch
>
> On Mon, Jul 27, 2009 at 2:33 PM, Gopal Gandhi <gopal.gandhi2008@yahoo.com
> >wrote:
>
> > I am going to apply a patch to hadoop (version 18.3). I searched on line
> > but could not find a step by step how-to manual. Would any hadoop guru
> tell
> > me how to apply a patch, say HADOOP-xxxx.patch to hadoop? Thanks lot.
> >
> >
> >
>
Re: Apply a patch to hadoop
Posted by Bill Graham <bi...@gmail.com>.
http://wiki.apache.org/hadoop/HowToContribute
Search for "Applying a patch" and you'll find this:
patch -p0 < cool_patch.patch
On Mon, Jul 27, 2009 at 2:33 PM, Gopal Gandhi <go...@yahoo.com>wrote:
> I am going to apply a patch to hadoop (version 18.3). I searched on line
> but could not find a step by step how-to manual. Would any hadoop guru tell
> me how to apply a patch, say HADOOP-xxxx.patch to hadoop? Thanks lot.
>
>
>
Apply a patch to hadoop
Posted by Gopal Gandhi <go...@yahoo.com>.
I am going to apply a patch to hadoop (version 18.3). I searched on line but could not find a step by step how-to manual. Would any hadoop guru tell me how to apply a patch, say HADOOP-xxxx.patch to hadoop? Thanks lot.
Re: Is there a way to configure rpc time out?
Posted by Mac Yuki <mc...@gmail.com>.
Just to clarify, is this because each individual job is taking a while to
complete and therefore being timed out? If this is your problem, you can
modify the property mapred.task.timeout in hadoop-default.xml. This is
described as:
The number of milliseconds before a task will be
terminated if it neither reads an input, writes an output, nor
updates its status string.
Hope that helps.
Mac
On Mon, Jul 27, 2009 at 1:48 PM, Steve Gao <st...@yahoo.com> wrote:
> Would anybody help with this simple question? Thanks lot
> -Steve
>
> --- On Tue, 7/21/09, Steve Gao <st...@yahoo.com> wrote:
>
> From: Steve Gao <st...@yahoo.com>
> Subject: Is there a way to configure rpc time out?
> To: common-user@hadoop.apache.org
> Cc: core-user@hadoop.apache.org
> Date: Tuesday, July 21, 2009, 5:46 PM
>
> My jobs in hadoop get lot of "rpc timeout" error when the cluster is busy.
> Indeed that's the what we don't want to see because the jobs were running
> fine, just need to wait a little long to process.
>
> Is there a way to configure rpc time out value so that it can wait longer?
> Thanks
>
>
>
>
>
>
>
>
Re: Is there a way to configure rpc time out?
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
What version are you running?
If you move to 0.18 or higher, you shouldn't see these.
Raghu.
Steve Gao wrote:
> Would anybody help with this simple question? Thanks lot
> -Steve
>
> --- On Tue, 7/21/09, Steve Gao <st...@yahoo.com> wrote:
>
> From: Steve Gao <st...@yahoo.com>
> Subject: Is there a way to configure rpc time out?
> To: common-user@hadoop.apache.org
> Cc: core-user@hadoop.apache.org
> Date: Tuesday, July 21, 2009, 5:46 PM
>
> My jobs in hadoop get lot of "rpc timeout" error when the cluster is busy. Indeed that's the what we don't want to see because the jobs were running fine, just need to wait a little long to process.
>
> Is there a way to configure rpc time out value so that it can wait longer? Thanks
>
>
>
>
>
>
>
Re: Is there a way to configure rpc time out?
Posted by Steve Gao <st...@yahoo.com>.
Would anybody help with this simple question? Thanks lot
-Steve
--- On Tue, 7/21/09, Steve Gao <st...@yahoo.com> wrote:
From: Steve Gao <st...@yahoo.com>
Subject: Is there a way to configure rpc time out?
To: common-user@hadoop.apache.org
Cc: core-user@hadoop.apache.org
Date: Tuesday, July 21, 2009, 5:46 PM
My jobs in hadoop get lot of "rpc timeout" error when the cluster is busy. Indeed that's the what we don't want to see because the jobs were running fine, just need to wait a little long to process.
Is there a way to configure rpc time out value so that it can wait longer? Thanks
Is there a way to configure rpc time out?
Posted by Steve Gao <st...@yahoo.com>.
My jobs in hadoop get lot of "rpc timeout" error when the cluster is busy. Indeed that's the what we don't want to see because the jobs were running fine, just need to wait a little long to process.
Is there a way to configure rpc time out value so that it can wait longer? Thanks
Hadoop logging
Posted by Arv Mistry <ar...@kindsight.net>.
Hi,
Recently we ran out of disk space on the hadoop machine, and on
investigation we found it was the hadoop log4j logs.
In the log4j.properties file I have set the hadoop.root.logger=ERROR
yet I still the daily hadoop-hadoopadmin-*.log files with INFO level
logging in them. These never seem to get trimmed or rollover.
Does anyone know how to limit ALL hadoop logs?
Do I have to set each daemon logging individually? i.e.
log4j.logger.org.apache.hadoop.mapred.JobTracker=ERROR etc
Cheers Arv
JobTracker not started [ipc.RemoteException] when starting Hadoop 0.20.0 cluster on Amazon EC2 using contrib/ec2 scripts
Posted by Jeyendran Balakrishnan <jb...@docomolabs-usa.com>.
Hello,
I downloaded Hadoop 0.20.0 and used the src/contrib/ec2/bin scripts to
launch a Hadoop cluster on Amazon EC2. To do so, I modified the bundled
scripts above for my EC2 account, and then created my own Hadoop 0.20.0
AMI. The steps I followed for creating AMIs and launching EC2 Hadoop
clusters are the same I was using for over a year with Hadoop 0.18.* and
0.19.*.
I launched an instance with my new Hadoop 0.20.0 AMI, then logged in and
ran the following to launch a new cluster:
root(/vol/hadoop-0.20.0)> bin/launch-hadoop-cluster hadoop-test 2
After the usual EC2 wait, one master and two slave instances were
launched on EC2, as expected. When I ssh'ed into the instances, here is
what I found:
Slaves: DataNode and NameNode are running
Master: Only NameNode is running
I could use HDFS commands (using $HADOOP_HOME/bin/hadoop scripts)
without any problems, from both master and slaves. However, since
JobTracker is not running, I cannot run map-reduce jobs.
I verified that the port for fs.default.name is set to 50001, the port
for mapred.job.tracker is set to 50002 and that there are no other port
conflicts for these ports.
I checked the logs from /vol/hadoop-0.20.0/logs for the JobTracker,
reproduced below:
-----------------------------------------------
<<<
2009-07-20 16:56:30,273 WARN org.apache.hadoop.conf.Configuration:
DEPRECATED: hadoop-site.xml found in the classpath. Usage of
hadoop-site.xml is deprecated. Instead use core-site.xml,
mapred-site.xml and h
dfs-site.xml to override properties of core-default.xml,
mapred-default.xml and hdfs-default.xml respectively
2009-07-20 16:56:30,320 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = domU-12-31-39-04-30-16/10.240.55.228
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.0
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r
763504; compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
************************************************************/
2009-07-20 16:56:31,332 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=JobTracker, port=50002
2009-07-20 16:56:31,603 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2009-07-20 16:56:31,900 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50030
2009-07-20 16:56:31,900 INFO org.mortbay.log: jetty-6.1.14
2009-07-20 16:56:33,461 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2009-07-20 16:56:33,462 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=JobTracker, sessionId=
2009-07-20 16:56:33,531 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker up at: 50002
2009-07-20 16:56:33,532 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker webserver: 50030
2009-07-20 16:56:51,554 INFO org.apache.hadoop.mapred.JobTracker:
Cleaning up the system directory
2009-07-20 16:56:53,060 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/mnt/hadoop/mapred/system/jobtracker.info could only be replicated to 0
nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
SNamesystem.java:1256)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:4
22)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo
cationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation
Handler.java:59)
at $Proxy4.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DF
SClient.java:2873)
...
...
2009-07-20 16:56:55,878 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/mnt/hadoop/mapred/system/jobtracker.info retries left 1
2009-07-20 16:56:59,082 WARN org.apache.hadoop.hdfs.DFSClient:
DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File /mnt/hadoop/mapred/system/jobtracker.info
could only
replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
SNamesystem.java:1256)
...
...
2009-07-20 16:57:00,092 FATAL org.apache.hadoop.mapred.JobTracker:
java.net.BindException: Problem binding to
domU-12-31-39-04-30-16.compute-1.internal/10.240.55.228:50002 : Address
already in use
at org.apache.hadoop.ipc.Server.bind(Server.java:190)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1537)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:174)
at
org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3528)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119
)
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.apache.hadoop.ipc.Server.bind(Server.java:188)
... 7 more
2009-07-20 16:57:00,093 INFO org.apache.hadoop.mapred.JobTracker:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at
domU-12-31-39-04-30-16/10.240.55.228
************************************************************/
>>>
-----------------------------------------------
So it looks like the JobTracker launched, but then died trying to
replicate the jobtracker.info file to one or more slaves.
Would appreciate any help in this...
Thanks a lot,
jp
RE: Unable to start Hadoop mapred cluster on EC2 with Hadoop 0.20.0
Posted by Jeyendran Balakrishnan <jb...@docomolabs-usa.com>.
Hi Todd,
Thanks for the suggestions. I checked netstat -a on the master, and it
doesn't seem to indicate that port 50002 is in use by anybody:
--------------------------------
root@domU-12-31-39-04-30-16 (/vol/hadoop-0.20.0/)> netstat -a |more
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address
State
tcp 0 0 *:8649 *:*
LISTEN
tcp 0 0 *:8651 *:*
LISTEN
tcp 0 0 *:8652 *:*
LISTEN
tcp 0 0 localhost.localdomain:smtp *:*
LISTEN
tcp 0 0 localhost.localdomain:8649
localhost.localdomain:58683 TIME_WAIT
tcp 0 0 localhost.localdomain:8649
localhost.localdomain:58685 TIME_WAIT
tcp 0 0 localhost.localdomain:8649
localhost.localdomain:58684 TIME_WAIT
tcp 0 0 localhost.localdomain:8649
localhost.localdomain:58686 TIME_WAIT
tcp 0 0 *:http *:*
LISTEN
tcp 0 0 domU-12-31-39-04-30-1:50001 *:*
LISTEN
tcp 0 0 *:50070 *:*
LISTEN
tcp 0 0 *:ssh *:*
LISTEN
tcp 0 0 *:42207 *:*
LISTEN
tcp 0 48 domU-12-31-39-04-30-16.:ssh
key1.docomolabs-usa.c:19829 ESTABLISHED
tcp 0 0 domU-12-31-39-04-30-1:50001
domU-12-31-39-04-1E-6:56434 ESTABLISHED
tcp 0 0 domU-12-31-39-04-30-1:50001
domU-12-31-39-04-30-C:51812 ESTABLISHED
udp 0 0 *:bootpc *:*
udp 0 0 *:8649 *:*
udp 0 0 *:filenet-rpc *:*
udp 0 0 *:filenet-nch *:*
Active UNIX domain sockets (servers and established)
...
--------------------------------
Also, below is the hadoop-site.xml on the master [auto-generated by the
contrib/ec2 scripts and downloaded to the master and slaves]:
--------------------------------
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/mnt/hadoop</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://domU-12-31-39-04-30-16.compute-1.internal:50001</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://domU-12-31-39-04-30-16.compute-1.internal:50002</value>
</property>
<property>
<name>tasktracker.http.threads</name>
<value>80</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>3</value>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>3</value>
</property>
<property>
<name>mapred.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.output.compression.type</name>
<value>BLOCK</value>
</property>
<property>
<name>dfs.client.block.write.retries</name>
<value>3</value>
</property>
</configuration>
--------------------------------
which says that fs.default.name [name node] and mapred.job.tracker [job
tracker] are assigned to different ports [50001 and 50002 respectively].
Would appreciate any thoughts...
Thanks,
Jeyendran
-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Monday, July 20, 2009 2:31 PM
To: common-user@hadoop.apache.org
Subject: Re: Unable to start Hadoop mapred cluster on EC2 with Hadoop
0.20.0
Hi Jeyendran,
Is it possible that you've configured the jobtracker's RPC address
(mapred.job.tracker) to be the same as its HTTP address? The "Address
already in use" error indicates that someone is already claiming port
50002.
That might be another daemon on the same machine, or it could be another
port in use by the JT.
-Todd
On Mon, Jul 20, 2009 at 2:20 PM, Jeyendran Balakrishnan <
jbalakrishnan@docomolabs-usa.com> wrote:
> Hello,
>
> I downloaded Hadoop 0.20.0 and used the src/contrib/ec2/bin scripts to
> launch a Hadoop cluster on Amazon EC2. To do so, I modified the
bundled
> scripts above for my EC2 account, and then created my own Hadoop
0.20.0
> AMI. The steps I followed for creating AMIs and launching EC2 Hadoop
> clusters are the same I was using for over a year with Hadoop 0.18.*
and
> 0.19.*.
>
> I launched an instance with my new Hadoop 0.20.0 AMI, then logged in
and
> ran the following to launch a new cluster:
> root(/vol/hadoop-0.20.0)> bin/launch-hadoop-cluster hadoop-test 2
>
> After the usual EC2 wait, one master and two slave instances were
> launched on EC2, as expected. When I ssh'ed into the instances, here
is
> what I found:
>
> Slaves: DataNode and NameNode are running
> Master: Only NameNode is running
>
> I could use HDFS commands (using $HADOOP_HOME/bin/hadoop scripts)
> without any problems, from both master and slaves. However, since
> JobTracker is not running, I cannot run map-reduce jobs.
>
> I checked the logs from /vol/hadoop-0.20.0/logs for the JobTracker,
> reproduced below:
> -----------------------------------------------
> <<<
> 2009-07-20 16:56:30,273 WARN org.apache.hadoop.conf.Configuration:
> DEPRECATED: hadoop-site.xml found in the classpath. Usage of
> hadoop-site.xml is deprecated. Instead use core-site.xml,
> mapred-site.xml and h
> dfs-site.xml to override properties of core-default.xml,
> mapred-default.xml and hdfs-default.xml respectively
> 2009-07-20 16:56:30,320 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = domU-12-31-39-04-30-16/10.240.55.228
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.0
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r
> 763504; compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
> ************************************************************/
> 2009-07-20 16:56:31,332 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=JobTracker, port=50002
> 2009-07-20 16:56:31,603 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2009-07-20 16:56:31,900 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50030
> 2009-07-20 16:56:31,900 INFO org.mortbay.log: jetty-6.1.14
> 2009-07-20 16:56:33,461 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50030
> 2009-07-20 16:56:33,462 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=JobTracker, sessionId=
> 2009-07-20 16:56:33,531 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker up at: 50002
> 2009-07-20 16:56:33,532 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker webserver: 50030
> 2009-07-20 16:56:51,554 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning up the system directory
> 2009-07-20 16:56:53,060 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/jobtracker.info could only be replicated to
0
> nodes, instead of 1
> at
>
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
> SNamesystem.java:1256)
> at
>
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:4
> 22)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo
> cationHandler.java:82)
> at
>
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation
> Handler.java:59)
> at $Proxy4.addBlock(Unknown Source)
> at
>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DF
> SClient.java:2873)
> ...
> ...
> 2009-07-20 16:56:55,878 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/jobtracker.info retries left 1
> 2009-07-20 16:56:59,082 WARN org.apache.hadoop.hdfs.DFSClient:
> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: File /mnt/hadoop/mapred/system/jobtracker.info
> could only
> replicated to 0 nodes, instead of 1
> at
>
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
> SNamesystem.java:1256)
> ...
> ...
>
> 2009-07-20 16:57:00,092 FATAL org.apache.hadoop.mapred.JobTracker:
> java.net.BindException: Problem binding to
> domU-12-31-39-04-30-16.compute-1.internal/10.240.55.228:50002 :
Address
> already in use
> at org.apache.hadoop.ipc.Server.bind(Server.java:190)
> at
org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
> at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
> at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
> at
> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1537)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:174)
> at
> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3528)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind(Native Method)
> at
>
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119
> )
> at
> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at org.apache.hadoop.ipc.Server.bind(Server.java:188)
> ... 7 more
>
>
> 2009-07-20 16:57:00,093 INFO org.apache.hadoop.mapred.JobTracker:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down JobTracker at
> domU-12-31-39-04-30-16/10.240.55.228
> ************************************************************/
> >>>
> -----------------------------------------------
>
> So it looks like the JobTracker launched, but then died trying to
> replicate the jobtracker.info file to one or more slaves.
>
> Would appreciate any help in this...
>
> Thanks a lot,
> jp
>
>
Re: Unable to start Hadoop mapred cluster on EC2 with Hadoop 0.20.0
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Jeyendran,
Is it possible that you've configured the jobtracker's RPC address
(mapred.job.tracker) to be the same as its HTTP address? The "Address
already in use" error indicates that someone is already claiming port 50002.
That might be another daemon on the same machine, or it could be another
port in use by the JT.
-Todd
On Mon, Jul 20, 2009 at 2:20 PM, Jeyendran Balakrishnan <
jbalakrishnan@docomolabs-usa.com> wrote:
> Hello,
>
> I downloaded Hadoop 0.20.0 and used the src/contrib/ec2/bin scripts to
> launch a Hadoop cluster on Amazon EC2. To do so, I modified the bundled
> scripts above for my EC2 account, and then created my own Hadoop 0.20.0
> AMI. The steps I followed for creating AMIs and launching EC2 Hadoop
> clusters are the same I was using for over a year with Hadoop 0.18.* and
> 0.19.*.
>
> I launched an instance with my new Hadoop 0.20.0 AMI, then logged in and
> ran the following to launch a new cluster:
> root(/vol/hadoop-0.20.0)> bin/launch-hadoop-cluster hadoop-test 2
>
> After the usual EC2 wait, one master and two slave instances were
> launched on EC2, as expected. When I ssh'ed into the instances, here is
> what I found:
>
> Slaves: DataNode and NameNode are running
> Master: Only NameNode is running
>
> I could use HDFS commands (using $HADOOP_HOME/bin/hadoop scripts)
> without any problems, from both master and slaves. However, since
> JobTracker is not running, I cannot run map-reduce jobs.
>
> I checked the logs from /vol/hadoop-0.20.0/logs for the JobTracker,
> reproduced below:
> -----------------------------------------------
> <<<
> 2009-07-20 16:56:30,273 WARN org.apache.hadoop.conf.Configuration:
> DEPRECATED: hadoop-site.xml found in the classpath. Usage of
> hadoop-site.xml is deprecated. Instead use core-site.xml,
> mapred-site.xml and h
> dfs-site.xml to override properties of core-default.xml,
> mapred-default.xml and hdfs-default.xml respectively
> 2009-07-20 16:56:30,320 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = domU-12-31-39-04-30-16/10.240.55.228
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.0
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r
> 763504; compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
> ************************************************************/
> 2009-07-20 16:56:31,332 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=JobTracker, port=50002
> 2009-07-20 16:56:31,603 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2009-07-20 16:56:31,900 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50030
> 2009-07-20 16:56:31,900 INFO org.mortbay.log: jetty-6.1.14
> 2009-07-20 16:56:33,461 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50030
> 2009-07-20 16:56:33,462 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=JobTracker, sessionId=
> 2009-07-20 16:56:33,531 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker up at: 50002
> 2009-07-20 16:56:33,532 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker webserver: 50030
> 2009-07-20 16:56:51,554 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning up the system directory
> 2009-07-20 16:56:53,060 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/jobtracker.info could only be replicated to 0
> nodes, instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
> SNamesystem.java:1256)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:4
> 22)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo
> cationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation
> Handler.java:59)
> at $Proxy4.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DF
> SClient.java:2873)
> ...
> ...
> 2009-07-20 16:56:55,878 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/jobtracker.info retries left 1
> 2009-07-20 16:56:59,082 WARN org.apache.hadoop.hdfs.DFSClient:
> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: File /mnt/hadoop/mapred/system/jobtracker.info
> could only
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
> SNamesystem.java:1256)
> ...
> ...
>
> 2009-07-20 16:57:00,092 FATAL org.apache.hadoop.mapred.JobTracker:
> java.net.BindException: Problem binding to
> domU-12-31-39-04-30-16.compute-1.internal/10.240.55.228:50002 : Address
> already in use
> at org.apache.hadoop.ipc.Server.bind(Server.java:190)
> at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
> at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
> at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
> at
> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1537)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:174)
> at
> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3528)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119
> )
> at
> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at org.apache.hadoop.ipc.Server.bind(Server.java:188)
> ... 7 more
>
>
> 2009-07-20 16:57:00,093 INFO org.apache.hadoop.mapred.JobTracker:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down JobTracker at
> domU-12-31-39-04-30-16/10.240.55.228
> ************************************************************/
> >>>
> -----------------------------------------------
>
> So it looks like the JobTracker launched, but then died trying to
> replicate the jobtracker.info file to one or more slaves.
>
> Would appreciate any help in this...
>
> Thanks a lot,
> jp
>
>
RE: Unable to start Hadoop mapred cluster on EC2 with Hadoop 0.20.0
Posted by Jeyendran Balakrishnan <jb...@docomolabs-usa.com>.
Sorry, a typo in the previous post:
What I found running were:
Slaves: DataNode and TaskTracker are running
Master: Only NameNode is running
So everything except JobTracker on master are running.
Thanks,
jp
-----Original Message-----
From: Jeyendran Balakrishnan [mailto:jbalakrishnan@docomolabs-usa.com]
Sent: Monday, July 20, 2009 2:20 PM
To: common-user@hadoop.apache.org
Subject: Unable to start Hadoop mapred cluster on EC2 with Hadoop 0.20.0
Hello,
I downloaded Hadoop 0.20.0 and used the src/contrib/ec2/bin scripts to
launch a Hadoop cluster on Amazon EC2. To do so, I modified the bundled
scripts above for my EC2 account, and then created my own Hadoop 0.20.0
AMI. The steps I followed for creating AMIs and launching EC2 Hadoop
clusters are the same I was using for over a year with Hadoop 0.18.* and
0.19.*.
I launched an instance with my new Hadoop 0.20.0 AMI, then logged in and
ran the following to launch a new cluster:
root(/vol/hadoop-0.20.0)> bin/launch-hadoop-cluster hadoop-test 2
After the usual EC2 wait, one master and two slave instances were
launched on EC2, as expected. When I ssh'ed into the instances, here is
what I found:
Slaves: DataNode and NameNode are running
Master: Only NameNode is running
I could use HDFS commands (using $HADOOP_HOME/bin/hadoop scripts)
without any problems, from both master and slaves. However, since
JobTracker is not running, I cannot run map-reduce jobs.
I checked the logs from /vol/hadoop-0.20.0/logs for the JobTracker,
reproduced below:
-----------------------------------------------
<<<
2009-07-20 16:56:30,273 WARN org.apache.hadoop.conf.Configuration:
DEPRECATED: hadoop-site.xml found in the classpath. Usage of
hadoop-site.xml is deprecated. Instead use core-site.xml,
mapred-site.xml and h
dfs-site.xml to override properties of core-default.xml,
mapred-default.xml and hdfs-default.xml respectively
2009-07-20 16:56:30,320 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = domU-12-31-39-04-30-16/10.240.55.228
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.0
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r
763504; compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
************************************************************/
2009-07-20 16:56:31,332 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=JobTracker, port=50002
2009-07-20 16:56:31,603 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2009-07-20 16:56:31,900 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50030
2009-07-20 16:56:31,900 INFO org.mortbay.log: jetty-6.1.14
2009-07-20 16:56:33,461 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2009-07-20 16:56:33,462 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=JobTracker, sessionId=
2009-07-20 16:56:33,531 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker up at: 50002
2009-07-20 16:56:33,532 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker webserver: 50030
2009-07-20 16:56:51,554 INFO org.apache.hadoop.mapred.JobTracker:
Cleaning up the system directory
2009-07-20 16:56:53,060 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/mnt/hadoop/mapred/system/jobtracker.info could only be replicated to 0
nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
SNamesystem.java:1256)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:4
22)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo
cationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation
Handler.java:59)
at $Proxy4.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DF
SClient.java:2873)
...
...
2009-07-20 16:56:55,878 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/mnt/hadoop/mapred/system/jobtracker.info retries left 1
2009-07-20 16:56:59,082 WARN org.apache.hadoop.hdfs.DFSClient:
DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File /mnt/hadoop/mapred/system/jobtracker.info
could only
replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
SNamesystem.java:1256)
...
...
2009-07-20 16:57:00,092 FATAL org.apache.hadoop.mapred.JobTracker:
java.net.BindException: Problem binding to
domU-12-31-39-04-30-16.compute-1.internal/10.240.55.228:50002 : Address
already in use
at org.apache.hadoop.ipc.Server.bind(Server.java:190)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1537)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:174)
at
org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3528)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119
)
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.apache.hadoop.ipc.Server.bind(Server.java:188)
... 7 more
2009-07-20 16:57:00,093 INFO org.apache.hadoop.mapred.JobTracker:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at
domU-12-31-39-04-30-16/10.240.55.228
************************************************************/
>>>
-----------------------------------------------
So it looks like the JobTracker launched, but then died trying to
replicate the jobtracker.info file to one or more slaves.
Would appreciate any help in this...
Thanks a lot,
jp
Unable to start Hadoop mapred cluster on EC2 with Hadoop 0.20.0
Posted by Jeyendran Balakrishnan <jb...@docomolabs-usa.com>.
Hello,
I downloaded Hadoop 0.20.0 and used the src/contrib/ec2/bin scripts to
launch a Hadoop cluster on Amazon EC2. To do so, I modified the bundled
scripts above for my EC2 account, and then created my own Hadoop 0.20.0
AMI. The steps I followed for creating AMIs and launching EC2 Hadoop
clusters are the same I was using for over a year with Hadoop 0.18.* and
0.19.*.
I launched an instance with my new Hadoop 0.20.0 AMI, then logged in and
ran the following to launch a new cluster:
root(/vol/hadoop-0.20.0)> bin/launch-hadoop-cluster hadoop-test 2
After the usual EC2 wait, one master and two slave instances were
launched on EC2, as expected. When I ssh'ed into the instances, here is
what I found:
Slaves: DataNode and NameNode are running
Master: Only NameNode is running
I could use HDFS commands (using $HADOOP_HOME/bin/hadoop scripts)
without any problems, from both master and slaves. However, since
JobTracker is not running, I cannot run map-reduce jobs.
I checked the logs from /vol/hadoop-0.20.0/logs for the JobTracker,
reproduced below:
-----------------------------------------------
<<<
2009-07-20 16:56:30,273 WARN org.apache.hadoop.conf.Configuration:
DEPRECATED: hadoop-site.xml found in the classpath. Usage of
hadoop-site.xml is deprecated. Instead use core-site.xml,
mapred-site.xml and h
dfs-site.xml to override properties of core-default.xml,
mapred-default.xml and hdfs-default.xml respectively
2009-07-20 16:56:30,320 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = domU-12-31-39-04-30-16/10.240.55.228
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.0
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r
763504; compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
************************************************************/
2009-07-20 16:56:31,332 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=JobTracker, port=50002
2009-07-20 16:56:31,603 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2009-07-20 16:56:31,900 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50030
2009-07-20 16:56:31,900 INFO org.mortbay.log: jetty-6.1.14
2009-07-20 16:56:33,461 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2009-07-20 16:56:33,462 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=JobTracker, sessionId=
2009-07-20 16:56:33,531 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker up at: 50002
2009-07-20 16:56:33,532 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker webserver: 50030
2009-07-20 16:56:51,554 INFO org.apache.hadoop.mapred.JobTracker:
Cleaning up the system directory
2009-07-20 16:56:53,060 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/mnt/hadoop/mapred/system/jobtracker.info could only be replicated to 0
nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
SNamesystem.java:1256)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:4
22)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo
cationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation
Handler.java:59)
at $Proxy4.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DF
SClient.java:2873)
...
...
2009-07-20 16:56:55,878 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/mnt/hadoop/mapred/system/jobtracker.info retries left 1
2009-07-20 16:56:59,082 WARN org.apache.hadoop.hdfs.DFSClient:
DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File /mnt/hadoop/mapred/system/jobtracker.info
could only
replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
SNamesystem.java:1256)
...
...
2009-07-20 16:57:00,092 FATAL org.apache.hadoop.mapred.JobTracker:
java.net.BindException: Problem binding to
domU-12-31-39-04-30-16.compute-1.internal/10.240.55.228:50002 : Address
already in use
at org.apache.hadoop.ipc.Server.bind(Server.java:190)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1537)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:174)
at
org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3528)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119
)
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.apache.hadoop.ipc.Server.bind(Server.java:188)
... 7 more
2009-07-20 16:57:00,093 INFO org.apache.hadoop.mapred.JobTracker:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at
domU-12-31-39-04-30-16/10.240.55.228
************************************************************/
>>>
-----------------------------------------------
So it looks like the JobTracker launched, but then died trying to
replicate the jobtracker.info file to one or more slaves.
Would appreciate any help in this...
Thanks a lot,
jp
RE: Recovery following disk full
Posted by Arv Mistry <ar...@kindsight.net>.
Thanks guys, I did have a secondary name node running, so was able to recover from last checkpoint as Todd had suggested.
Cheers Arv
________________________________
From: Jakob Homan [mailto:jhoman@yahoo-inc.com]
Sent: Mon 20/07/2009 4:53 PM
To: common-user@hadoop.apache.org
Subject: Re: Recovery following disk full
The oiv handles the fsimage fil but not the edits log, so it wouldn't
help in this case. There has been talk about writing a similar tool for
the edits log but nothing has been decided.
Also, while the oiv will be included in 21, it works on images back to
18 (and maybe earlier). It's standalone, so it doesn't need a cluster
or anything, just the fsimage file.
Option c will be very tricky and earns its place as the last-ditch effort.
-jg
Tom White wrote:
> Is this an area where the Offline Image Viewer might be able to help
> in the future? It's not available for 0.18.3, but seems like it would
> be possible to extend it as a tool to help with c) in Todd's
> description.
>
> Tom
>
> On Mon, Jul 20, 2009 at 8:30 PM, Todd Lipcon<to...@cloudera.com> wrote:
>> Hi Arv,
>>
>> It sounds like your edits log in dfs.name.dir is corrupted since one of its
>> records got cut off by the disk filling up. When trying to replay the edit
>> log, it tries to read the entirety of that record and hits the end of file
>> unexpectedly - hence the EOFException.
>>
>> Your options at this point are:
>>
>> a) If you have a second copy of dfs.name.dir, it should also have a second
>> "edits" file. If it's longer it's possible that that copy is not corrupted.
>> I'd back up both copies, then duplicate the longer edit log into both name
>> dirs and try to start the namenode.
>>
>> b) If you were running a secondary namenode, you should have a checkpoint of
>> the fsimage from a few hours before the failure. You can recover the fsimage
>> from there. You'll lose some time period's worth of metadata edits, but you
>> should be able to get the FS running again.
>>
>> c) Last ditch attempt is to attempt to truncate the edit log at the correct
>> offset such that you avoid the EOFException. To do this would probably
>> involve adding some logging statements to the FSEditLog replay so you can
>> see what the byte offset of the last record it's trying to read is, and then
>> truncating the edit log right before that offset. This is somewhat
>> complicated and I wouldn't attempt it unless you (a) really need the data
>> and (b) don't have any other option.
>>
>> -Todd
>>
>> On Mon, Jul 20, 2009 at 12:27 PM, Arv Mistry <ar...@kindsight.net> wrote:
>>
>>> Hi,
>>>
>>> I'm getting the following error in starting up the namenode.
>>>
>>> What happened was one of our disks filled up, we reclaimed the
>>> disk space and tried to restart the hadoop daemons but the name node
>>> is now not starting up.
>>>
>>> Does anybody have any clues how to recover from this? I've tried
>>> searching through the Jira reports but nothing obvious.
>>>
>>> Appreciate any input, thanks.
>>>
>>> Cheers Arv
>>>
>>> 2009-07-20 14:57:41,712 INFO org.apache.hadoop.dfs.NameNode:
>>> STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting NameNode
>>> STARTUP_MSG: host = qa-cs1/192.168.0.54
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 0.18.3-dev
>>> STARTUP_MSG: build = -r ; compiled by 'bamboo' on Mon Nov 10 15:58:40
>>> PST 2008
>>> ************************************************************/
>>> 2009-07-20 14:57:41,801 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>>> Initializing RPC Metrics with hostName=NameNode, port=9000
>>> 2009-07-20 14:57:41,805 INFO org.apache.hadoop.dfs.NameNode: Namenode up
>>> at: 192.168.0.54/192.168.0.54:9000
>>> 2009-07-20 <http://192.168.0.54/192.168.0.54:9000%0A2009-07-20>14:57:41,808 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>>> Initializing JVM Metrics with processName=NameNode, sessionId=null
>>> 2009-07-20 14:57:41,816 INFO org.apache.hadoop.dfs.NameNodeMetrics:
>>> Initializing NameNodeMeterics using context
>>> object:org.apache.hadoop.metrics.spi.NullContext
>>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>>> fsOwner=hadoopadmin,hadoopadmin
>>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>>> supergroup=supergroup
>>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>>> isPermissionEnabled=true
>>> 2009-07-20 14:57:41,877 INFO org.apache.hadoop.dfs.FSNamesystemMetrics:
>>> Initializing FSNamesystemMeterics using context
>>> object:org.apache.hadoop.metrics.spi.NullContext
>>> 2009-07-20 14:57:41,878 INFO org.apache.hadoop.fs.FSNamesystem:
>>> Registered FSNamesystemStatusMBean
>>> 2009-07-20 14:57:41,908 INFO org.apache.hadoop.dfs.Storage: Number of
>>> files = 1808
>>> 2009-07-20 14:57:42,153 INFO org.apache.hadoop.dfs.Storage: Number of
>>> files under construction = 1
>>> 2009-07-20 14:57:42,157 INFO org.apache.hadoop.dfs.Storage: Image file
>>> of size 256399 loaded in 0 seconds.
>>> 2009-07-20 14:57:42,167 ERROR
>>> org.apache.hadoop.dfs.LeaseManager:
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605290.data
>>> not found in lease.paths
>>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605294.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>>> 90720_180000_1248113605451.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605464.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>>> 2009-07-20 14:57:42,167 ERROR
>>> org.apache.hadoop.dfs.LeaseManager:
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605294.data
>>> not found in lease.paths
>>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605451.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>>> 90720_180000_1248113605464.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>>> 2009-07-20 14:57:42,169 ERROR
>>> org.apache.hadoop.dfs.LeaseManager:
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data
>>> not found in lease.paths
>>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605290.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605294.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>>> 90720_180000_1248113605451.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605464.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>>> 2009-07-20 14:57:42,169 ERROR
>>> org.apache.hadoop.dfs.LeaseManager:
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data
>>> not found in lease.paths
>>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605290.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605294.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605451.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>>> 90720_180000_1248113605464.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>>> 2009-07-20 14:57:42,171 ERROR org.apache.hadoop.fs.FSNamesystem:
>>> FSNamesystem initialization failed.
>>> java.io.EOFException
>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
>>> at org.apache.hadoop.dfs.FSImage.readString(FSImage.java:1368)
>>> at
>>> org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:447)
>>> at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:846)
>>> at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:675)
>>> at
>>> org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:289)
>>> at
>>> org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)
>>> at
>>> org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:296)
>>> at
>>> org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:275)
>>>
>>>
>
Re: Recovery following disk full
Posted by Jakob Homan <jh...@yahoo-inc.com>.
The oiv handles the fsimage fil but not the edits log, so it wouldn't
help in this case. There has been talk about writing a similar tool for
the edits log but nothing has been decided.
Also, while the oiv will be included in 21, it works on images back to
18 (and maybe earlier). It's standalone, so it doesn't need a cluster
or anything, just the fsimage file.
Option c will be very tricky and earns its place as the last-ditch effort.
-jg
Tom White wrote:
> Is this an area where the Offline Image Viewer might be able to help
> in the future? It's not available for 0.18.3, but seems like it would
> be possible to extend it as a tool to help with c) in Todd's
> description.
>
> Tom
>
> On Mon, Jul 20, 2009 at 8:30 PM, Todd Lipcon<to...@cloudera.com> wrote:
>> Hi Arv,
>>
>> It sounds like your edits log in dfs.name.dir is corrupted since one of its
>> records got cut off by the disk filling up. When trying to replay the edit
>> log, it tries to read the entirety of that record and hits the end of file
>> unexpectedly - hence the EOFException.
>>
>> Your options at this point are:
>>
>> a) If you have a second copy of dfs.name.dir, it should also have a second
>> "edits" file. If it's longer it's possible that that copy is not corrupted.
>> I'd back up both copies, then duplicate the longer edit log into both name
>> dirs and try to start the namenode.
>>
>> b) If you were running a secondary namenode, you should have a checkpoint of
>> the fsimage from a few hours before the failure. You can recover the fsimage
>> from there. You'll lose some time period's worth of metadata edits, but you
>> should be able to get the FS running again.
>>
>> c) Last ditch attempt is to attempt to truncate the edit log at the correct
>> offset such that you avoid the EOFException. To do this would probably
>> involve adding some logging statements to the FSEditLog replay so you can
>> see what the byte offset of the last record it's trying to read is, and then
>> truncating the edit log right before that offset. This is somewhat
>> complicated and I wouldn't attempt it unless you (a) really need the data
>> and (b) don't have any other option.
>>
>> -Todd
>>
>> On Mon, Jul 20, 2009 at 12:27 PM, Arv Mistry <ar...@kindsight.net> wrote:
>>
>>> Hi,
>>>
>>> I'm getting the following error in starting up the namenode.
>>>
>>> What happened was one of our disks filled up, we reclaimed the
>>> disk space and tried to restart the hadoop daemons but the name node
>>> is now not starting up.
>>>
>>> Does anybody have any clues how to recover from this? I've tried
>>> searching through the Jira reports but nothing obvious.
>>>
>>> Appreciate any input, thanks.
>>>
>>> Cheers Arv
>>>
>>> 2009-07-20 14:57:41,712 INFO org.apache.hadoop.dfs.NameNode:
>>> STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting NameNode
>>> STARTUP_MSG: host = qa-cs1/192.168.0.54
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 0.18.3-dev
>>> STARTUP_MSG: build = -r ; compiled by 'bamboo' on Mon Nov 10 15:58:40
>>> PST 2008
>>> ************************************************************/
>>> 2009-07-20 14:57:41,801 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>>> Initializing RPC Metrics with hostName=NameNode, port=9000
>>> 2009-07-20 14:57:41,805 INFO org.apache.hadoop.dfs.NameNode: Namenode up
>>> at: 192.168.0.54/192.168.0.54:9000
>>> 2009-07-20 <http://192.168.0.54/192.168.0.54:9000%0A2009-07-20>14:57:41,808 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>>> Initializing JVM Metrics with processName=NameNode, sessionId=null
>>> 2009-07-20 14:57:41,816 INFO org.apache.hadoop.dfs.NameNodeMetrics:
>>> Initializing NameNodeMeterics using context
>>> object:org.apache.hadoop.metrics.spi.NullContext
>>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>>> fsOwner=hadoopadmin,hadoopadmin
>>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>>> supergroup=supergroup
>>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>>> isPermissionEnabled=true
>>> 2009-07-20 14:57:41,877 INFO org.apache.hadoop.dfs.FSNamesystemMetrics:
>>> Initializing FSNamesystemMeterics using context
>>> object:org.apache.hadoop.metrics.spi.NullContext
>>> 2009-07-20 14:57:41,878 INFO org.apache.hadoop.fs.FSNamesystem:
>>> Registered FSNamesystemStatusMBean
>>> 2009-07-20 14:57:41,908 INFO org.apache.hadoop.dfs.Storage: Number of
>>> files = 1808
>>> 2009-07-20 14:57:42,153 INFO org.apache.hadoop.dfs.Storage: Number of
>>> files under construction = 1
>>> 2009-07-20 14:57:42,157 INFO org.apache.hadoop.dfs.Storage: Image file
>>> of size 256399 loaded in 0 seconds.
>>> 2009-07-20 14:57:42,167 ERROR
>>> org.apache.hadoop.dfs.LeaseManager:
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605290.data
>>> not found in lease.paths
>>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605294.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>>> 90720_180000_1248113605451.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605464.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>>> 2009-07-20 14:57:42,167 ERROR
>>> org.apache.hadoop.dfs.LeaseManager:
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605294.data
>>> not found in lease.paths
>>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605451.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>>> 90720_180000_1248113605464.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>>> 2009-07-20 14:57:42,169 ERROR
>>> org.apache.hadoop.dfs.LeaseManager:
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data
>>> not found in lease.paths
>>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605290.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605294.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>>> 90720_180000_1248113605451.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605464.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>>> 2009-07-20 14:57:42,169 ERROR
>>> org.apache.hadoop.dfs.LeaseManager:
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data
>>> not found in lease.paths
>>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605290.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605294.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605451.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>>> 90720_180000_1248113605464.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>>> 2009-07-20 14:57:42,171 ERROR org.apache.hadoop.fs.FSNamesystem:
>>> FSNamesystem initialization failed.
>>> java.io.EOFException
>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
>>> at org.apache.hadoop.dfs.FSImage.readString(FSImage.java:1368)
>>> at
>>> org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:447)
>>> at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:846)
>>> at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:675)
>>> at
>>> org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:289)
>>> at
>>> org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)
>>> at
>>> org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:296)
>>> at
>>> org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:275)
>>>
>>>
>
Re: Recovery following disk full
Posted by Tom White <to...@cloudera.com>.
Is this an area where the Offline Image Viewer might be able to help
in the future? It's not available for 0.18.3, but seems like it would
be possible to extend it as a tool to help with c) in Todd's
description.
Tom
On Mon, Jul 20, 2009 at 8:30 PM, Todd Lipcon<to...@cloudera.com> wrote:
> Hi Arv,
>
> It sounds like your edits log in dfs.name.dir is corrupted since one of its
> records got cut off by the disk filling up. When trying to replay the edit
> log, it tries to read the entirety of that record and hits the end of file
> unexpectedly - hence the EOFException.
>
> Your options at this point are:
>
> a) If you have a second copy of dfs.name.dir, it should also have a second
> "edits" file. If it's longer it's possible that that copy is not corrupted.
> I'd back up both copies, then duplicate the longer edit log into both name
> dirs and try to start the namenode.
>
> b) If you were running a secondary namenode, you should have a checkpoint of
> the fsimage from a few hours before the failure. You can recover the fsimage
> from there. You'll lose some time period's worth of metadata edits, but you
> should be able to get the FS running again.
>
> c) Last ditch attempt is to attempt to truncate the edit log at the correct
> offset such that you avoid the EOFException. To do this would probably
> involve adding some logging statements to the FSEditLog replay so you can
> see what the byte offset of the last record it's trying to read is, and then
> truncating the edit log right before that offset. This is somewhat
> complicated and I wouldn't attempt it unless you (a) really need the data
> and (b) don't have any other option.
>
> -Todd
>
> On Mon, Jul 20, 2009 at 12:27 PM, Arv Mistry <ar...@kindsight.net> wrote:
>
>> Hi,
>>
>> I'm getting the following error in starting up the namenode.
>>
>> What happened was one of our disks filled up, we reclaimed the
>> disk space and tried to restart the hadoop daemons but the name node
>> is now not starting up.
>>
>> Does anybody have any clues how to recover from this? I've tried
>> searching through the Jira reports but nothing obvious.
>>
>> Appreciate any input, thanks.
>>
>> Cheers Arv
>>
>> 2009-07-20 14:57:41,712 INFO org.apache.hadoop.dfs.NameNode:
>> STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting NameNode
>> STARTUP_MSG: host = qa-cs1/192.168.0.54
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 0.18.3-dev
>> STARTUP_MSG: build = -r ; compiled by 'bamboo' on Mon Nov 10 15:58:40
>> PST 2008
>> ************************************************************/
>> 2009-07-20 14:57:41,801 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>> Initializing RPC Metrics with hostName=NameNode, port=9000
>> 2009-07-20 14:57:41,805 INFO org.apache.hadoop.dfs.NameNode: Namenode up
>> at: 192.168.0.54/192.168.0.54:9000
>> 2009-07-20 <http://192.168.0.54/192.168.0.54:9000%0A2009-07-20>14:57:41,808 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=NameNode, sessionId=null
>> 2009-07-20 14:57:41,816 INFO org.apache.hadoop.dfs.NameNodeMetrics:
>> Initializing NameNodeMeterics using context
>> object:org.apache.hadoop.metrics.spi.NullContext
>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>> fsOwner=hadoopadmin,hadoopadmin
>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>> supergroup=supergroup
>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>> isPermissionEnabled=true
>> 2009-07-20 14:57:41,877 INFO org.apache.hadoop.dfs.FSNamesystemMetrics:
>> Initializing FSNamesystemMeterics using context
>> object:org.apache.hadoop.metrics.spi.NullContext
>> 2009-07-20 14:57:41,878 INFO org.apache.hadoop.fs.FSNamesystem:
>> Registered FSNamesystemStatusMBean
>> 2009-07-20 14:57:41,908 INFO org.apache.hadoop.dfs.Storage: Number of
>> files = 1808
>> 2009-07-20 14:57:42,153 INFO org.apache.hadoop.dfs.Storage: Number of
>> files under construction = 1
>> 2009-07-20 14:57:42,157 INFO org.apache.hadoop.dfs.Storage: Image file
>> of size 256399 loaded in 0 seconds.
>> 2009-07-20 14:57:42,167 ERROR
>> org.apache.hadoop.dfs.LeaseManager:
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605290.data
>> not found in lease.paths
>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605294.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>> 90720_180000_1248113605451.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605464.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>> 2009-07-20 14:57:42,167 ERROR
>> org.apache.hadoop.dfs.LeaseManager:
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605294.data
>> not found in lease.paths
>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605451.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>> 90720_180000_1248113605464.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>> 2009-07-20 14:57:42,169 ERROR
>> org.apache.hadoop.dfs.LeaseManager:
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data
>> not found in lease.paths
>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605290.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605294.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>> 90720_180000_1248113605451.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605464.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>> 2009-07-20 14:57:42,169 ERROR
>> org.apache.hadoop.dfs.LeaseManager:
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data
>> not found in lease.paths
>> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605290.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605294.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605451.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
>> 90720_180000_1248113605464.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
>> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
>> 2009-07-20 14:57:42,171 ERROR org.apache.hadoop.fs.FSNamesystem:
>> FSNamesystem initialization failed.
>> java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
>> at org.apache.hadoop.dfs.FSImage.readString(FSImage.java:1368)
>> at
>> org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:447)
>> at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:846)
>> at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:675)
>> at
>> org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:289)
>> at
>> org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)
>> at
>> org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:296)
>> at
>> org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:275)
>>
>>
>
Re: Recovery following disk full
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Arv,
It sounds like your edits log in dfs.name.dir is corrupted since one of its
records got cut off by the disk filling up. When trying to replay the edit
log, it tries to read the entirety of that record and hits the end of file
unexpectedly - hence the EOFException.
Your options at this point are:
a) If you have a second copy of dfs.name.dir, it should also have a second
"edits" file. If it's longer it's possible that that copy is not corrupted.
I'd back up both copies, then duplicate the longer edit log into both name
dirs and try to start the namenode.
b) If you were running a secondary namenode, you should have a checkpoint of
the fsimage from a few hours before the failure. You can recover the fsimage
from there. You'll lose some time period's worth of metadata edits, but you
should be able to get the FS running again.
c) Last ditch attempt is to attempt to truncate the edit log at the correct
offset such that you avoid the EOFException. To do this would probably
involve adding some logging statements to the FSEditLog replay so you can
see what the byte offset of the last record it's trying to read is, and then
truncating the edit log right before that offset. This is somewhat
complicated and I wouldn't attempt it unless you (a) really need the data
and (b) don't have any other option.
-Todd
On Mon, Jul 20, 2009 at 12:27 PM, Arv Mistry <ar...@kindsight.net> wrote:
> Hi,
>
> I'm getting the following error in starting up the namenode.
>
> What happened was one of our disks filled up, we reclaimed the
> disk space and tried to restart the hadoop daemons but the name node
> is now not starting up.
>
> Does anybody have any clues how to recover from this? I've tried
> searching through the Jira reports but nothing obvious.
>
> Appreciate any input, thanks.
>
> Cheers Arv
>
> 2009-07-20 14:57:41,712 INFO org.apache.hadoop.dfs.NameNode:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = qa-cs1/192.168.0.54
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.18.3-dev
> STARTUP_MSG: build = -r ; compiled by 'bamboo' on Mon Nov 10 15:58:40
> PST 2008
> ************************************************************/
> 2009-07-20 14:57:41,801 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=NameNode, port=9000
> 2009-07-20 14:57:41,805 INFO org.apache.hadoop.dfs.NameNode: Namenode up
> at: 192.168.0.54/192.168.0.54:9000
> 2009-07-20 <http://192.168.0.54/192.168.0.54:9000%0A2009-07-20>14:57:41,808 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=NameNode, sessionId=null
> 2009-07-20 14:57:41,816 INFO org.apache.hadoop.dfs.NameNodeMetrics:
> Initializing NameNodeMeterics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
> fsOwner=hadoopadmin,hadoopadmin
> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
> supergroup=supergroup
> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
> isPermissionEnabled=true
> 2009-07-20 14:57:41,877 INFO org.apache.hadoop.dfs.FSNamesystemMetrics:
> Initializing FSNamesystemMeterics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2009-07-20 14:57:41,878 INFO org.apache.hadoop.fs.FSNamesystem:
> Registered FSNamesystemStatusMBean
> 2009-07-20 14:57:41,908 INFO org.apache.hadoop.dfs.Storage: Number of
> files = 1808
> 2009-07-20 14:57:42,153 INFO org.apache.hadoop.dfs.Storage: Number of
> files under construction = 1
> 2009-07-20 14:57:42,157 INFO org.apache.hadoop.dfs.Storage: Image file
> of size 256399 loaded in 0 seconds.
> 2009-07-20 14:57:42,167 ERROR
> org.apache.hadoop.dfs.LeaseManager:
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605290.data
> not found in lease.paths
> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605294.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
> 90720_180000_1248113605451.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605464.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
> 2009-07-20 14:57:42,167 ERROR
> org.apache.hadoop.dfs.LeaseManager:
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605294.data
> not found in lease.paths
> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605451.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
> 90720_180000_1248113605464.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
> 2009-07-20 14:57:42,169 ERROR
> org.apache.hadoop.dfs.LeaseManager:
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data
> not found in lease.paths
> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605290.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605294.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
> 90720_180000_1248113605451.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605464.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
> 2009-07-20 14:57:42,169 ERROR
> org.apache.hadoop.dfs.LeaseManager:
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data
> not found in lease.paths
> (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605290.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605294.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605451.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
> 90720_180000_1248113605464.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data,
> /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
> 2009-07-20 14:57:42,171 ERROR org.apache.hadoop.fs.FSNamesystem:
> FSNamesystem initialization failed.
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
> at org.apache.hadoop.dfs.FSImage.readString(FSImage.java:1368)
> at
> org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:447)
> at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:846)
> at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:675)
> at
> org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:289)
> at
> org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)
> at
> org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:296)
> at
> org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:275)
>
>
Recovery following disk full
Posted by Arv Mistry <ar...@kindsight.net>.
Hi,
I'm getting the following error in starting up the namenode.
What happened was one of our disks filled up, we reclaimed the
disk space and tried to restart the hadoop daemons but the name node
is now not starting up.
Does anybody have any clues how to recover from this? I've tried
searching through the Jira reports but nothing obvious.
Appreciate any input, thanks.
Cheers Arv
2009-07-20 14:57:41,712 INFO org.apache.hadoop.dfs.NameNode:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = qa-cs1/192.168.0.54
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.18.3-dev
STARTUP_MSG: build = -r ; compiled by 'bamboo' on Mon Nov 10 15:58:40
PST 2008
************************************************************/
2009-07-20 14:57:41,801 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=NameNode, port=9000
2009-07-20 14:57:41,805 INFO org.apache.hadoop.dfs.NameNode: Namenode up
at: 192.168.0.54/192.168.0.54:9000
2009-07-20 14:57:41,808 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=NameNode, sessionId=null
2009-07-20 14:57:41,816 INFO org.apache.hadoop.dfs.NameNodeMetrics:
Initializing NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
fsOwner=hadoopadmin,hadoopadmin
2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
supergroup=supergroup
2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
isPermissionEnabled=true
2009-07-20 14:57:41,877 INFO org.apache.hadoop.dfs.FSNamesystemMetrics:
Initializing FSNamesystemMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2009-07-20 14:57:41,878 INFO org.apache.hadoop.fs.FSNamesystem:
Registered FSNamesystemStatusMBean
2009-07-20 14:57:41,908 INFO org.apache.hadoop.dfs.Storage: Number of
files = 1808
2009-07-20 14:57:42,153 INFO org.apache.hadoop.dfs.Storage: Number of
files under construction = 1
2009-07-20 14:57:42,157 INFO org.apache.hadoop.dfs.Storage: Image file
of size 256399 loaded in 0 seconds.
2009-07-20 14:57:42,167 ERROR
org.apache.hadoop.dfs.LeaseManager: /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605290.data not found in lease.paths (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605294.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
90720_180000_1248113605451.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605464.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
2009-07-20 14:57:42,167 ERROR
org.apache.hadoop.dfs.LeaseManager: /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605294.data not found in lease.paths (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605451.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
90720_180000_1248113605464.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
2009-07-20 14:57:42,169 ERROR
org.apache.hadoop.dfs.LeaseManager: /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605298.data not found in lease.paths (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605290.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605294.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
90720_180000_1248113605451.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605464.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
2009-07-20 14:57:42,169 ERROR
org.apache.hadoop.dfs.LeaseManager: /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605303.data not found in lease.paths (=[/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605328.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605335.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605337.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605340.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605346.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_170000_1248113605401.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605290.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605294.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605432.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605451.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_200
90720_180000_1248113605464.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605487.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605499.data, /opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_180000_1248113605539.data])
2009-07-20 14:57:42,171 ERROR org.apache.hadoop.fs.FSNamesystem:
FSNamesystem initialization failed.
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
at org.apache.hadoop.dfs.FSImage.readString(FSImage.java:1368)
at
org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:447)
at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:846)
at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:675)
at
org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:289)
at
org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)
at
org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:296)
at
org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:275)
Re: Why /tmp directory?
Posted by Amareshwari Sriramadasu <am...@yahoo-inc.com>.
You would get "org.apache.hadoop.util.DiskChecker$DiskErrorException:
Could not find any valid local directory" if there is no space left on
disk to accomodate the file.
Thanks
Amareshwari
akhil1988 wrote:
> Hi All,
>
> I want to know why do we generally use tmp directory(and not any other) for
> storing hdfs data, knowing the fact that tmp directory is used for storing
> only temporary data?
>
> I was wondering this because when I run a Hbase job on large data, I get
> this DiskErrorExcpetion:
>
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for
>
> taskTracker/jobcache/job_200904072051_0001/attempt_200904072051_0001_m_000000_1/output/file.out
>
> Please, excuse if this is a naive question.
>
> --Akhil
>
>