You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Matthew John <tm...@gmail.com> on 2011/05/11 15:26:45 UTC
Bad connection to FS. command aborted
Hi all!
I have been trying to figure out why I m getting this error!
All that I did was :
1) Use a single node cluster
2) Made some modifications in the core (in some MapRed modules).
Successfully compiled it
3) Tried bin/start-dfs.sh alone.
All the required daemons (NN and DN) are up.
The NameNode and DataNode logs are nt showing any errors/exceptions.
Only interesting thing I found was :
*WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch
from 10.72.147.109:40048 got version 94 expected version 3 *
in the NameNode logs.
Someone please help me out of this!
Matthew
Re: Bad connection to FS. command aborted
Posted by Matthew John <tm...@gmail.com>.
I did the following:
*1) modified bin/hadoop*
*
*
*for f in $HADOOP_HOME/hadoop-*-core.jar; do*
* CLASSPATH=${CLASSPATH}:$f;*
*done*
*
*
*modified to *
*
*
*for f in $HADOOP_HOME/build/hadoop-*-core.jar; do*
* CLASSPATH=${CLASSPATH}:$f;*
*done*
This takes care of the version problem statement that used to come.
*2) The I got an error related to "Error register getProtocolVersion"*
I changed my hadoop-metrics back to NullContext settings.
*3) Still I am getting problem with IPC server. and the same error - Bad
connection to FS. Command aborted.*
Following is my Namenode log. Someone please help help out since I am not
very much familiar with the ipc server mechanisms around! :(
2011-05-11 19:34:30,114 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = mattHDFS1set1/10.72.147.109
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.3-dev
STARTUP_MSG: build = -r ; compiled by 'matthew' on Wed May 11 19:12:36
IST 2011
************************************************************/
2011-05-11 19:34:30,253 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=NameNode, port=54310
2011-05-11 19:34:30,261 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
mattHDFS1set1/10.72.147.109:54310
2011-05-11 19:34:30,271 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=NameNode, sessionId=null
2011-05-11 19:34:30,272 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing
NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2011-05-11 19:34:30,385 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
fsOwner=matthew,matthew,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
2011-05-11 19:34:30,385 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2011-05-11 19:34:30,385 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=true
2011-05-11 19:34:30,395 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
Initializing FSNamesystemMetrics using context
object:org.apache.hadoop.metrics.spi.NullContext
2011-05-11 19:34:30,397 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2011-05-11 19:34:30,449 INFO org.apache.hadoop.hdfs.server.common.Storage:
Number of files = 1
2011-05-11 19:34:30,455 INFO org.apache.hadoop.hdfs.server.common.Storage:
Number of files under construction = 0
2011-05-11 19:34:30,455 INFO org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 97 loaded in 0 seconds.
2011-05-11 19:34:30,455 INFO org.apache.hadoop.hdfs.server.common.Storage:
Edits file /mount_xfs/tmpdir/hadoop-matthew/dfs/name/current/edits of size 4
edits # 0 loaded in 0 seconds.
2011-05-11 19:34:30,488 INFO org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 97 saved in 0 seconds.
2011-05-11 19:34:30,535 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
FSImage in 204 msecs
2011-05-11 19:34:30,537 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks
= 0
2011-05-11 19:34:30,537 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid
blocks = 0
2011-05-11 19:34:30,537 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
under-replicated blocks = 0
2011-05-11 19:34:30,537 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
over-replicated blocks = 0
2011-05-11 19:34:30,537 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Leaving safe mode after 0 secs.
2011-05-11 19:34:30,537 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Network topology has 0 racks and 0 datanodes
2011-05-11 19:34:30,538 INFO org.apache.hadoop.hdfs.StateChange: STATE*
UnderReplicatedBlocks has 0 blocks
2011-05-11 19:34:36,080 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2011-05-11 19:34:36,163 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
Opening the listener on 50070
2011-05-11 19:34:36,165 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50070
webServer.getConnectors()[0].getLocalPort() returned 50070
2011-05-11 19:34:36,165 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50070
2011-05-11 19:34:36,165 INFO org.mortbay.log: jetty-6.1.14
2011-05-11 19:34:36,594 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50070
2011-05-11 19:34:36,594 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
0.0.0.0:50070
2011-05-11 19:34:36,604 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2011-05-11 19:34:36,609 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 54310: starting
2011-05-11 19:34:36,619 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 54310: starting
2011-05-11 19:34:36,622 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 54310: starting
2011-05-11 19:34:36,631 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 54310: starting
2011-05-11 19:34:36,639 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 54310: starting
2011-05-11 19:34:36,640 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 54310: starting
2011-05-11 19:34:36,655 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 54310: starting
2011-05-11 19:34:36,656 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 6 on 54310: starting
2011-05-11 19:34:36,658 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 7 on 54310: starting
2011-05-11 19:34:36,658 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 8 on 54310: starting
2011-05-11 19:34:36,669 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 54310: starting
2011-05-11 19:37:36,548 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.registerDatanode: node registration from
10.72.147.109:50010storage
DS-1515207802-10.72.147.109-50010-1305118592183
2011-05-11 19:37:36,551 INFO org.apache.hadoop.net.NetworkTopology: Adding a
new node: /default-rack/10.72.147.109:50010
Thanks,
Matthew
On Wed, May 11, 2011 at 7:13 PM, Habermaas, William <
William.Habermaas@fatwire.com> wrote:
> If the hadoop script is picking up a different hadoop-core jar then the
> classes that ipc to the NN will be using a different version.
>
> Bill
>
> -----Original Message-----
> From: Matthew John [mailto:tmatthewjohn1988@gmail.com]
> Sent: Wednesday, May 11, 2011 9:41 AM
> To: common-user@hadoop.apache.org
> Subject: Re: Bad connection to FS. command aborted
>
> I did a "ant jar" after modifying two files in the mapred module. This,
> from
> what I understand, creates a hadoop-*-core.jar in the build folder. Now I
> assume that will be used henceforth for any execution. So how can this be a
> problem if I am running a single-node cluster. Version mismatch with whom ?
>
> On Wed, May 11, 2011 at 7:07 PM, Habermaas, William <
> William.Habermaas@fatwire.com> wrote:
>
> > The Hadoop IPCs are version specific. That is done to prevent an older
> > version from talking to a newer one. Even if nothing has changed in the
> > internal protocols the version check is enforced. Make sure the new
> > hadoop-core.jar from your modification is on the classpath used by the
> > hadoop shell script.
> >
> > Bill
> >
> > -----Original Message-----
> > From: Matthew John [mailto:tmatthewjohn1988@gmail.com]
> > Sent: Wednesday, May 11, 2011 9:27 AM
> > To: common-user
> > Subject: Bad connection to FS. command aborted
> >
> > Hi all!
> >
> > I have been trying to figure out why I m getting this error!
> >
> > All that I did was :
> > 1) Use a single node cluster
> > 2) Made some modifications in the core (in some MapRed modules).
> > Successfully compiled it
> > 3) Tried bin/start-dfs.sh alone.
> >
> > All the required daemons (NN and DN) are up.
> > The NameNode and DataNode logs are nt showing any errors/exceptions.
> >
> > Only interesting thing I found was :
> > *WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch
> > from 10.72.147.109:40048 got version 94 expected version 3 *
> > in the NameNode logs.
> >
> > Someone please help me out of this!
> >
> > Matthew
> >
>
RE: Bad connection to FS. command aborted
Posted by "Habermaas, William" <Wi...@fatwire.com>.
If the hadoop script is picking up a different hadoop-core jar then the classes that ipc to the NN will be using a different version.
Bill
-----Original Message-----
From: Matthew John [mailto:tmatthewjohn1988@gmail.com]
Sent: Wednesday, May 11, 2011 9:41 AM
To: common-user@hadoop.apache.org
Subject: Re: Bad connection to FS. command aborted
I did a "ant jar" after modifying two files in the mapred module. This, from
what I understand, creates a hadoop-*-core.jar in the build folder. Now I
assume that will be used henceforth for any execution. So how can this be a
problem if I am running a single-node cluster. Version mismatch with whom ?
On Wed, May 11, 2011 at 7:07 PM, Habermaas, William <
William.Habermaas@fatwire.com> wrote:
> The Hadoop IPCs are version specific. That is done to prevent an older
> version from talking to a newer one. Even if nothing has changed in the
> internal protocols the version check is enforced. Make sure the new
> hadoop-core.jar from your modification is on the classpath used by the
> hadoop shell script.
>
> Bill
>
> -----Original Message-----
> From: Matthew John [mailto:tmatthewjohn1988@gmail.com]
> Sent: Wednesday, May 11, 2011 9:27 AM
> To: common-user
> Subject: Bad connection to FS. command aborted
>
> Hi all!
>
> I have been trying to figure out why I m getting this error!
>
> All that I did was :
> 1) Use a single node cluster
> 2) Made some modifications in the core (in some MapRed modules).
> Successfully compiled it
> 3) Tried bin/start-dfs.sh alone.
>
> All the required daemons (NN and DN) are up.
> The NameNode and DataNode logs are nt showing any errors/exceptions.
>
> Only interesting thing I found was :
> *WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch
> from 10.72.147.109:40048 got version 94 expected version 3 *
> in the NameNode logs.
>
> Someone please help me out of this!
>
> Matthew
>
Re: Bad connection to FS. command aborted
Posted by Matthew John <tm...@gmail.com>.
I did a "ant jar" after modifying two files in the mapred module. This, from
what I understand, creates a hadoop-*-core.jar in the build folder. Now I
assume that will be used henceforth for any execution. So how can this be a
problem if I am running a single-node cluster. Version mismatch with whom ?
On Wed, May 11, 2011 at 7:07 PM, Habermaas, William <
William.Habermaas@fatwire.com> wrote:
> The Hadoop IPCs are version specific. That is done to prevent an older
> version from talking to a newer one. Even if nothing has changed in the
> internal protocols the version check is enforced. Make sure the new
> hadoop-core.jar from your modification is on the classpath used by the
> hadoop shell script.
>
> Bill
>
> -----Original Message-----
> From: Matthew John [mailto:tmatthewjohn1988@gmail.com]
> Sent: Wednesday, May 11, 2011 9:27 AM
> To: common-user
> Subject: Bad connection to FS. command aborted
>
> Hi all!
>
> I have been trying to figure out why I m getting this error!
>
> All that I did was :
> 1) Use a single node cluster
> 2) Made some modifications in the core (in some MapRed modules).
> Successfully compiled it
> 3) Tried bin/start-dfs.sh alone.
>
> All the required daemons (NN and DN) are up.
> The NameNode and DataNode logs are nt showing any errors/exceptions.
>
> Only interesting thing I found was :
> *WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch
> from 10.72.147.109:40048 got version 94 expected version 3 *
> in the NameNode logs.
>
> Someone please help me out of this!
>
> Matthew
>
RE: Bad connection to FS. command aborted
Posted by "Habermaas, William" <Wi...@fatwire.com>.
The Hadoop IPCs are version specific. That is done to prevent an older version from talking to a newer one. Even if nothing has changed in the internal protocols the version check is enforced. Make sure the new hadoop-core.jar from your modification is on the classpath used by the hadoop shell script.
Bill
-----Original Message-----
From: Matthew John [mailto:tmatthewjohn1988@gmail.com]
Sent: Wednesday, May 11, 2011 9:27 AM
To: common-user
Subject: Bad connection to FS. command aborted
Hi all!
I have been trying to figure out why I m getting this error!
All that I did was :
1) Use a single node cluster
2) Made some modifications in the core (in some MapRed modules).
Successfully compiled it
3) Tried bin/start-dfs.sh alone.
All the required daemons (NN and DN) are up.
The NameNode and DataNode logs are nt showing any errors/exceptions.
Only interesting thing I found was :
*WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch
from 10.72.147.109:40048 got version 94 expected version 3 *
in the NameNode logs.
Someone please help me out of this!
Matthew
Re: Bad connection to FS. command aborted
Posted by Eric Fiala <er...@fiala.ca>.
Matthew - this normally means there's a mismatch version of hadoop somewhere
in your pipe - I'd suggest to make sure that all your hadoop-core-x-y-z.jar
are the same, across the board.
EF
On Wed, May 11, 2011 at 7:26 AM, Matthew John <tm...@gmail.com>wrote:
> Hi all!
>
> I have been trying to figure out why I m getting this error!
>
> All that I did was :
> 1) Use a single node cluster
> 2) Made some modifications in the core (in some MapRed modules).
> Successfully compiled it
> 3) Tried bin/start-dfs.sh alone.
>
> All the required daemons (NN and DN) are up.
> The NameNode and DataNode logs are nt showing any errors/exceptions.
>
> Only interesting thing I found was :
> *WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch
> from 10.72.147.109:40048 got version 94 expected version 3 *
> in the NameNode logs.
>
> Someone please help me out of this!
>
> Matthew
>