You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ashishgandhe <as...@gmail.com> on 2008/09/28 01:14:58 UTC
Re: bad connection ack with firstbadlink 10.23.....
Hi Chandra,
Were you able to resolve this error? I'm facing the exact same issue.
Thanks,
Ashish
chandravadana wrote:
>
>
> hi..
>
> i'm using a cluster of 3 systems.. 1 being the master and the other 2
> are slaves.
>
> i don get any errors when i cluster 2 systems. when i add the 3rd one, i
> get the following error..
>
> master: 10.232.25.197
> slave1: 10.232.25.96
> slave2:10.232.25.69
>
> wordcount/hi/ dir is the input dir
>
> when i execute :
>
> # bin/hadoop dfs -copyFromLocal wordcount/hi wordcount/ins
>
> 08/08/25 13:43:30 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:30 INFO dfs.DFSClient: Abandoning block
> blk_-3916191835981679734
> 08/08/25 13:43:36 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:36 INFO dfs.DFSClient: Abandoning block
> blk_-7058774921272589893
> 08/08/25 13:43:42 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:42 INFO dfs.DFSClient: Abandoning block
> blk_3767065959322874247
> 08/08/25 13:43:48 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:48 INFO dfs.DFSClient: Abandoning block
> blk_-8330992315825789947
> 08/08/25 13:43:54 WARN dfs.DFSClient: DataStreamer Exception:
> java.io.IOException: Unable to create new block.
> 08/08/25 13:43:54 WARN dfs.DFSClient: Error Recovery for block
> blk_-8330992315825789947 bad datanode[1]
> copyFromLocal: Could not get block locations. Aborting...
>
> when i examine the log file of the slave, i see this
>
> 2008-08-25 13:42:18,140 INFO org.apache.hadoop.dfs.DataNode:
> STARTUP_MSG: /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = slave1/10.232.25.96 STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.16.4 STARTUP_MSG: build =
> http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r
> 652614; compiled by 'hadoopqa' on Fri May 2 00:18:12 UTC 2008
> ************************************************************/
> 2008-08-25 13:42:18,634 INFO org.apache.hadoop.dfs.Storage: Storage
> directory /etc/hadoop_install/hadoop-0.16.4/datanodedir is not
> formatted.
> 2009-08-25 13:42:18,634 INFO org.apache.hadoop.dfs.Storage:
> Formatting ...
> 2008-08-25 13:42:18,701 INFO org.apache.hadoop.dfs.DataNode: Registered
> FSDatasetStatusMBean
> 2008-08-25 13:42:18,701 INFO org.apache.hadoop.dfs.DataNode: Opened
> server at 50010
> 2008-08-25 13:42:18,705 INFO org.apache.hadoop.dfs.DataNode: Balancing
> bandwith is 1048576 bytes/s
> 2008-08-25 13:42:18,911 INFO org.mortbay.util.Credential: Checking
> Resource aliases
> 2008-08-25 13:42:19,013 INFO org.mortbay.http.HttpServer: Version
> Jetty/5.1.4 2008-08-25 13:42:19,014 INFO org.mortbay.util.Container:
> Started HttpContext[/static,/static]
> 2008-08-25 13:42:19,014 INFO org.mortbay.util.Container: Started
> HttpContext[/logs,/logs]
> 2008-08-25 13:42:19,579 INFO org.mortbay.util.Container: Started
> org.mortbay.jetty.servlet.WebApplicationHandler@11ff436
> 2008-08-25 13:42:19,658 INFO org.mortbay.util.Container: Started
> WebApplicationContext[/,/]
> 2008-08-25 13:42:19,661 INFO org.mortbay.http.SocketListener: Started
> SocketListener on 0.0.0.0:50075
> 2008-08-25 13:42:19,661 INFO org.mortbay.util.Container: Started
> org.mortbay.jetty.Server@1b8f864
> 2008-08-25 13:42:19,706 INFO org.apache.hadoop.dfs.DataNode: New storage
> id DS-860242092-10.232.25.96-50010-1219651939700 is assigned to data-
> node 10.232.25.96:50010
> 2008-08-25 13:42:19,733 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=DataNode, sessionId=null
> 2008-08-25 13:42:19,755 INFO org.apache.hadoop.dfs.DataNode:
> 10.232.25.96:50010In DataNode.run, data = FSDataset
> {dirpath='/etc/hadoop_install/hadoop-0.16.4/datanodedir/current'}
> 2008-08-25 13:42:19,755 INFO org.apache.hadoop.dfs.DataNode: using
> BLOCKREPORT_INTERVAL of 3538776msec Initial delay: 60000msec
> 2008-08-25 13:42:19,828 INFO org.apache.hadoop.dfs.DataNode: BlockReport
> of 0 blocks got processed in 20 msecs
> 2008-08-25 13:45:43,982 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_1031802361447574775 src: /10.232.25.197:40282
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,032 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,081 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_1031802361447574775 of size 3161 from /10.232.25.197
> 2008-08-25 13:45:44,081 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_1031802361447574775 terminating
> 2008-08-25 13:45:44,105 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_-1924738157193733587 src: /10.232.25.197:40285
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,106 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,193 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_-1924738157193733587 of size 6628 from /10.232.25.197
> 2008-08-25 13:45:44,193 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_-1924738157193733587 terminating
> 2008-08-25 13:45:44,212 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_7001275375373078911 src: /10.232.25.197:40287
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,213 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 008-08-25 13:45:44,256 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_7001275375373078911 of size 3161 from /10.232.25.197
> 2008-08-25 13:45:44,256 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_7001275375373078911 terminating
> 2008-08-25 13:45:44,277 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_-7471693146363669981 src: /10.232.25.197:40289
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,278 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,362 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_-7471693146363669981 of size 6628 from /10.232.25.197
> 2008-08-25 13:45:44,362 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_-7471693146363669981 terminating
> 2008-08-25 13:45:44,380 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_-6619078097753318750 src: /10.232.25.197:40291
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,380 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,424 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_-6619078097753318750 of size 2778 from /10.232.25.197
> 2008-08-25 13:45:44,424 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_-6619078097753318750 terminating
> 2008-08-25 13:45:44,440 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_1527614673854389960 src: /10.232.25.197:40293
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,441 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,526 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_1527614673854389960 of size 4616 from /10.232.25.197
> 2008-08-25 13:45:44,526 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_1527614673854389960 terminating
> 2008-08-25 13:47:21,331 INFO org.apache.hadoop.dfs.DataBlockScanner:
> Verification succeeded for blk_1527614673854389960
> 2008-08-25 13:48:11,458 INFO org.apache.hadoop.dfs.DataBlockScanner:
> Verification succeeded for blk_7001275375373078911
>
>
> i should i do.. kindly help me in resolving this issue...
> Thanks in advance..
>
>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information.
> If you are not the intended recipient, please contact the sender by reply
> e-mail and destroy all copies of the original message.
> Any unauthorized review, use, disclosure, dissemination, forwarding,
> printing or copying of this email or any action taken in reliance on this
> e-mail is strictly
> prohibited and may be unlawful.
>
>
--
View this message in context: http://www.nabble.com/bad-connection-ack-with-firstbadlink-10.23.....-tp19140032p19707301.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: bad connection ack with firstbadlink 10.23.....
Posted by chandravadana <Ch...@cognizant.com>.
ashishgandhe wrote:
>
> Hi Chandra,
>
> Were you able to resolve this error? I'm facing the exact same issue.
>
> hi..
> yes was able to fix this.. it was a firewall issue... try disabling
> firewall on all nodes in the cluster...
>
> Thanks,
> S.Chandravadana
>
> Thanks,
> Ashish
>
>
>
> chandravadana wrote:
>>
>>
>> hi..
>>
>> i'm using a cluster of 3 systems.. 1 being the master and the other 2
>> are slaves.
>>
>> i don get any errors when i cluster 2 systems. when i add the 3rd one, i
>> get the following error..
>>
>> master: 10.232.25.197
>> slave1: 10.232.25.96
>> slave2:10.232.25.69
>>
>> wordcount/hi/ dir is the input dir
>>
>> when i execute :
>>
>> # bin/hadoop dfs -copyFromLocal wordcount/hi wordcount/ins
>>
>> 08/08/25 13:43:30 INFO dfs.DFSClient: Exception in
>> createBlockOutputStream java.io.IOException: Bad connect ack with
>> firstBadLink 10.232.25.69:50010
>> 08/08/25 13:43:30 INFO dfs.DFSClient: Abandoning block
>> blk_-3916191835981679734
>> 08/08/25 13:43:36 INFO dfs.DFSClient: Exception in
>> createBlockOutputStream java.io.IOException: Bad connect ack with
>> firstBadLink 10.232.25.69:50010
>> 08/08/25 13:43:36 INFO dfs.DFSClient: Abandoning block
>> blk_-7058774921272589893
>> 08/08/25 13:43:42 INFO dfs.DFSClient: Exception in
>> createBlockOutputStream java.io.IOException: Bad connect ack with
>> firstBadLink 10.232.25.69:50010
>> 08/08/25 13:43:42 INFO dfs.DFSClient: Abandoning block
>> blk_3767065959322874247
>> 08/08/25 13:43:48 INFO dfs.DFSClient: Exception in
>> createBlockOutputStream java.io.IOException: Bad connect ack with
>> firstBadLink 10.232.25.69:50010
>> 08/08/25 13:43:48 INFO dfs.DFSClient: Abandoning block
>> blk_-8330992315825789947
>> 08/08/25 13:43:54 WARN dfs.DFSClient: DataStreamer Exception:
>> java.io.IOException: Unable to create new block.
>> 08/08/25 13:43:54 WARN dfs.DFSClient: Error Recovery for block
>> blk_-8330992315825789947 bad datanode[1]
>> copyFromLocal: Could not get block locations. Aborting...
>>
>> when i examine the log file of the slave, i see this
>>
>> 2008-08-25 13:42:18,140 INFO org.apache.hadoop.dfs.DataNode:
>> STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting DataNode
>> STARTUP_MSG: host = slave1/10.232.25.96 STARTUP_MSG: args = []
>> STARTUP_MSG: version = 0.16.4 STARTUP_MSG: build =
>> http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r
>> 652614; compiled by 'hadoopqa' on Fri May 2 00:18:12 UTC 2008
>> ************************************************************/
>> 2008-08-25 13:42:18,634 INFO org.apache.hadoop.dfs.Storage: Storage
>> directory /etc/hadoop_install/hadoop-0.16.4/datanodedir is not
>> formatted.
>> 2009-08-25 13:42:18,634 INFO org.apache.hadoop.dfs.Storage:
>> Formatting ...
>> 2008-08-25 13:42:18,701 INFO org.apache.hadoop.dfs.DataNode: Registered
>> FSDatasetStatusMBean
>> 2008-08-25 13:42:18,701 INFO org.apache.hadoop.dfs.DataNode: Opened
>> server at 50010
>> 2008-08-25 13:42:18,705 INFO org.apache.hadoop.dfs.DataNode: Balancing
>> bandwith is 1048576 bytes/s
>> 2008-08-25 13:42:18,911 INFO org.mortbay.util.Credential: Checking
>> Resource aliases
>> 2008-08-25 13:42:19,013 INFO org.mortbay.http.HttpServer: Version
>> Jetty/5.1.4 2008-08-25 13:42:19,014 INFO org.mortbay.util.Container:
>> Started HttpContext[/static,/static]
>> 2008-08-25 13:42:19,014 INFO org.mortbay.util.Container: Started
>> HttpContext[/logs,/logs]
>> 2008-08-25 13:42:19,579 INFO org.mortbay.util.Container: Started
>> org.mortbay.jetty.servlet.WebApplicationHandler@11ff436
>> 2008-08-25 13:42:19,658 INFO org.mortbay.util.Container: Started
>> WebApplicationContext[/,/]
>> 2008-08-25 13:42:19,661 INFO org.mortbay.http.SocketListener: Started
>> SocketListener on 0.0.0.0:50075
>> 2008-08-25 13:42:19,661 INFO org.mortbay.util.Container: Started
>> org.mortbay.jetty.Server@1b8f864
>> 2008-08-25 13:42:19,706 INFO org.apache.hadoop.dfs.DataNode: New storage
>> id DS-860242092-10.232.25.96-50010-1219651939700 is assigned to data-
>> node 10.232.25.96:50010
>> 2008-08-25 13:42:19,733 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=DataNode, sessionId=null
>> 2008-08-25 13:42:19,755 INFO org.apache.hadoop.dfs.DataNode:
>> 10.232.25.96:50010In DataNode.run, data = FSDataset
>> {dirpath='/etc/hadoop_install/hadoop-0.16.4/datanodedir/current'}
>> 2008-08-25 13:42:19,755 INFO org.apache.hadoop.dfs.DataNode: using
>> BLOCKREPORT_INTERVAL of 3538776msec Initial delay: 60000msec
>> 2008-08-25 13:42:19,828 INFO org.apache.hadoop.dfs.DataNode: BlockReport
>> of 0 blocks got processed in 20 msecs
>> 2008-08-25 13:45:43,982 INFO org.apache.hadoop.dfs.DataNode: Receiving
>> block blk_1031802361447574775 src: /10.232.25.197:40282
>> dest: /10.232.25.197:50010
>> 2008-08-25 13:45:44,032 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
>> forwarding connect ack to upstream firstbadlink is
>> 2008-08-25 13:45:44,081 INFO org.apache.hadoop.dfs.DataNode: Received
>> block blk_1031802361447574775 of size 3161 from /10.232.25.197
>> 2008-08-25 13:45:44,081 INFO org.apache.hadoop.dfs.DataNode:
>> PacketResponder 0 for block blk_1031802361447574775 terminating
>> 2008-08-25 13:45:44,105 INFO org.apache.hadoop.dfs.DataNode: Receiving
>> block blk_-1924738157193733587 src: /10.232.25.197:40285
>> dest: /10.232.25.197:50010
>> 2008-08-25 13:45:44,106 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
>> forwarding connect ack to upstream firstbadlink is
>> 2008-08-25 13:45:44,193 INFO org.apache.hadoop.dfs.DataNode: Received
>> block blk_-1924738157193733587 of size 6628 from /10.232.25.197
>> 2008-08-25 13:45:44,193 INFO org.apache.hadoop.dfs.DataNode:
>> PacketResponder 0 for block blk_-1924738157193733587 terminating
>> 2008-08-25 13:45:44,212 INFO org.apache.hadoop.dfs.DataNode: Receiving
>> block blk_7001275375373078911 src: /10.232.25.197:40287
>> dest: /10.232.25.197:50010
>> 2008-08-25 13:45:44,213 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
>> forwarding connect ack to upstream firstbadlink is
>> 008-08-25 13:45:44,256 INFO org.apache.hadoop.dfs.DataNode: Received
>> block blk_7001275375373078911 of size 3161 from /10.232.25.197
>> 2008-08-25 13:45:44,256 INFO org.apache.hadoop.dfs.DataNode:
>> PacketResponder 0 for block blk_7001275375373078911 terminating
>> 2008-08-25 13:45:44,277 INFO org.apache.hadoop.dfs.DataNode: Receiving
>> block blk_-7471693146363669981 src: /10.232.25.197:40289
>> dest: /10.232.25.197:50010
>> 2008-08-25 13:45:44,278 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
>> forwarding connect ack to upstream firstbadlink is
>> 2008-08-25 13:45:44,362 INFO org.apache.hadoop.dfs.DataNode: Received
>> block blk_-7471693146363669981 of size 6628 from /10.232.25.197
>> 2008-08-25 13:45:44,362 INFO org.apache.hadoop.dfs.DataNode:
>> PacketResponder 0 for block blk_-7471693146363669981 terminating
>> 2008-08-25 13:45:44,380 INFO org.apache.hadoop.dfs.DataNode: Receiving
>> block blk_-6619078097753318750 src: /10.232.25.197:40291
>> dest: /10.232.25.197:50010
>> 2008-08-25 13:45:44,380 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
>> forwarding connect ack to upstream firstbadlink is
>> 2008-08-25 13:45:44,424 INFO org.apache.hadoop.dfs.DataNode: Received
>> block blk_-6619078097753318750 of size 2778 from /10.232.25.197
>> 2008-08-25 13:45:44,424 INFO org.apache.hadoop.dfs.DataNode:
>> PacketResponder 0 for block blk_-6619078097753318750 terminating
>> 2008-08-25 13:45:44,440 INFO org.apache.hadoop.dfs.DataNode: Receiving
>> block blk_1527614673854389960 src: /10.232.25.197:40293
>> dest: /10.232.25.197:50010
>> 2008-08-25 13:45:44,441 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
>> forwarding connect ack to upstream firstbadlink is
>> 2008-08-25 13:45:44,526 INFO org.apache.hadoop.dfs.DataNode: Received
>> block blk_1527614673854389960 of size 4616 from /10.232.25.197
>> 2008-08-25 13:45:44,526 INFO org.apache.hadoop.dfs.DataNode:
>> PacketResponder 0 for block blk_1527614673854389960 terminating
>> 2008-08-25 13:47:21,331 INFO org.apache.hadoop.dfs.DataBlockScanner:
>> Verification succeeded for blk_1527614673854389960
>> 2008-08-25 13:48:11,458 INFO org.apache.hadoop.dfs.DataBlockScanner:
>> Verification succeeded for blk_7001275375373078911
>>
>>
>> i should i do.. kindly help me in resolving this issue...
>> Thanks in advance..
>>
>>
>>
>> This e-mail and any files transmitted with it are for the sole use of the
>> intended recipient(s) and may contain confidential and privileged
>> information.
>> If you are not the intended recipient, please contact the sender by reply
>> e-mail and destroy all copies of the original message.
>> Any unauthorized review, use, disclosure, dissemination, forwarding,
>> printing or copying of this email or any action taken in reliance on this
>> e-mail is strictly
>> prohibited and may be unlawful.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/bad-connection-ack-with-firstbadlink-10.23.....-tp19140032p20028948.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.