You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Christian Stucchio <st...@gmail.com> on 2011/02/20 15:37:56 UTC

Hadoop fsck failure

Greetings,

I recently had a power failure, resulting in all my servers shutting
down. Everything appears to have recovered, but I am now unable to run
fsck:

$ hadoop fsck /
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 503 for URL: http://0.0.0.0:50070/fsck?path=%2F
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
	at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:123)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:159)

Any ideas as to what is going on? My data appears safe, and everything
else seems to be working fine.

Chris

Re: Hadoop fsck failure

Posted by Christian Stucchio <st...@gmail.com>.
It's definitely uninitialized. From the namenode log:

2011-02-20 22:38:31,789 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2011-02-20 22:38:31,835 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open()
is -1. Opening the listener on 50070
2011-02-20 22:38:31,836 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50070
webServer.getConnectors()[0].getLocalPort() returned 50070
2011-02-20 22:38:31,836 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50070
2011-02-20 22:38:31,836 INFO org.mortbay.log: jetty-6.1.14
2011-02-20 22:38:31,932 WARN org.mortbay.log: Failed startup of
context org.mortbay.jetty.webapp.WebAppContext@7c19f9d2{/,file:/usr/local/hadoop/webapps/hdfs}
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:114)
	at java.util.jar.JarFile.<init>(JarFile.java:135)
	at java.util.jar.JarFile.<init>(JarFile.java:99)
	at org.mortbay.jetty.webapp.TagLibConfiguration.configureWebApp(TagLibConfiguration.java:168)
	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1231)
	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:460)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
	at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
	at org.mortbay.jetty.Server.doStart(Server.java:222)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.apache.hadoop.http.HttpServer.start(HttpServer.java:461)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:246)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:202)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
2011-02-20 22:38:31,939 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50070

Googling this error only found this page, which suggests that the
JVM's cache of jar files might have been modified somehow. I don't
fully understand the error.

http://docs.codehaus.org/display/JETTY/ZIP+exception+reading+jar+or+war+files

However, on the theory that something was wrong with a jar file
somewhere, I reinstalled the hadoop jars on my server. This resolved
the issue.


Thanks for your help,



Chris



On Sun, Feb 20, 2011 at 11:47 AM, Harsh J <qw...@gmail.com> wrote:
> Hello,
>
> There could be something wrong with the Jetty webserver that starts
> along with a daemon (Used by the Reducer to fetch map outputs, and
> seemingly also by fsck from NN's server). Perhaps the logs (of NN, TT)
> can tell you more on what may have happened when it was initializing
> the Jetty HTTP service. Look at the initializing part (early up in the
> logs, during start-up).
>
> Quite possibly, it may be down/uninitialized due to some reason. Are
> you able to view the Web UIs of each service (services that publish
> one)?
>
> --
> Harsh J
> www.harshj.com
>

Re: Hadoop fsck failure

Posted by Harsh J <qw...@gmail.com>.
Hello,

There could be something wrong with the Jetty webserver that starts
along with a daemon (Used by the Reducer to fetch map outputs, and
seemingly also by fsck from NN's server). Perhaps the logs (of NN, TT)
can tell you more on what may have happened when it was initializing
the Jetty HTTP service. Look at the initializing part (early up in the
logs, during start-up).

Quite possibly, it may be down/uninitialized due to some reason. Are
you able to view the Web UIs of each service (services that publish
one)?

-- 
Harsh J
www.harshj.com

Re: Hadoop fsck failure

Posted by Christian Stucchio <st...@gmail.com>.
Just a followup to this question.

After further experimentation, I'm also running into errors in the
task tracker. While running a job, I see:

...
11/02/20 21:59:08 INFO mapred.JobClient:  map 93% reduce 0%
11/02/20 22:00:05 INFO mapred.JobClient:  map 100% reduce 0%
11/02/20 22:03:59 INFO mapred.JobClient: Task Id :
attempt_201102202150_0001_m_000001_0, Status : FAILED
Too many fetch-failures
11/02/20 22:03:59 WARN mapred.JobClient: Error reading task
outputServer returned HTTP response code: 503 for URL:
http://stylewok-hadoop:50060/tasklog?plaintext=true&taskid=attempt_201102202150_0001_m_000001_0&filter=stdout
11/02/20 22:03:59 WARN mapred.JobClient: Error reading task
outputServer returned HTTP response code: 503 for URL:
http://stylewok-hadoop:50060/tasklog?plaintext=true&taskid=attempt_201102202150_0001_m_000001_0&filter=stderr
11/02/20 22:04:06 INFO mapred.JobClient:  map 93% reduce 0%

Any help with this would be greatly appreciated.


Chris



On Sun, Feb 20, 2011 at 9:37 AM, Christian Stucchio <st...@gmail.com> wrote:
> Greetings,
>
> I recently had a power failure, resulting in all my servers shutting
> down. Everything appears to have recovered, but I am now unable to run
> fsck:
>
> $ hadoop fsck /
> Exception in thread "main" java.io.IOException: Server returned HTTP
> response code: 503 for URL: http://0.0.0.0:50070/fsck?path=%2F
>        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
>        at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:123)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>        at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:159)
>
> Any ideas as to what is going on? My data appears safe, and everything
> else seems to be working fine.
>
> Chris
>