You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Weihua JIANG <we...@gmail.com> on 2011/11/01 08:28:37 UTC

Data node problem after reinstall

Hi all,

I am not sure whether it is a hbase problem or a hdfs problem.

When have 8 datanodes & regionservers each with 12 disks. One data node is
down due to its system disk broken. After replacing the disk and reinstall
the OS, we tried to online this DN & region server. The region server is
OK. But, the data node seems can only accept block replication from other
data nodes, but failed to accept writing from HBase. Even after we removed
all content of data dir and restarted the DN, this problem still exists.
The log is:
2011-11-01 22:39:44,397 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
blk_-6413273237261014407_77967 src: /10.1.2.23:56859 dest:
/10.1.2.25:50010of size 67108864
2011-11-01 22:39:45,959 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
blk_-6561707380479467846_78302 src: /10.1.2.20:47578 dest:
/10.1.2.25:50010of size 67108864
2011-11-01 22:39:46,314 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-6233096255264073662_235292 src: /10.1.2.26:36311 dest: /10.1.2.25:50010
2011-11-01 22:39:46,829 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.1.2.21:35450, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
cliID: DFSClient_1189047403, offset: 0, srvID:
DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
blk_7603159785320952833_235458, duration: 20790059066
2011-11-01 22:39:46,829 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
block blk_7603159785320952833_235458 terminating
2011-11-01 22:39:46,850 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_7980572561428708688_235458 src: /10.1.2.19:37630 dest: /10.1.2.25:50010
2011-11-01 22:39:47,093 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.1.2.19:37611, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
cliID: DFSClient_-81251145, offset: 0, srvID:
DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
blk_-9067419333696936066_235458, duration: 20478973800
2011-11-01 22:39:47,093 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
block blk_-9067419333696936066_235458 terminating
2011-11-01 22:39:47,117 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-495608789656244385_235458 src: /10.1.2.21:35468 dest: /10.1.2.25:50010
2011-11-01 22:39:47,185 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.1.2.23:56845, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
cliID: DFSClient_1647768037, offset: 0, srvID:
DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
blk_-7935479347615843321_235458, duration: 20595122675
2011-11-01 22:39:47,185 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
block blk_-7935479347615843321_235458 terminating
2011-11-01 22:39:47,325 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
blk_-6233096255264073662_235292 src: /10.1.2.26:36311 dest:
/10.1.2.25:50010of size 67108864
2011-11-01 22:39:49,981 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.1.2.25:60888, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
cliID: DFSClient_-1345645880, offset: 0, srvID:
DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
blk_2614344148027463064_235458, duration: 19952693395
2011-11-01 22:39:49,982 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for
block blk_2614344148027463064_235458 terminating
2011-11-01 22:39:49,999 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_5157459941530714338_235458 src: /10.1.2.25:60892 dest: /10.1.2.25:50010
2011-11-01 22:39:51,258 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-6108054911705086653_39962 src: /10.1.2.21:35475 dest: /10.1.2.25:50010
2011-11-01 22:39:51,259 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
blk_-6108054911705086653_39962 src: /10.1.2.21:35475 dest:
/10.1.2.25:50010of size 124
2011-11-01 22:39:52,357 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-5980257649446708968_78270 src: /10.1.2.23:56867 dest: /10.1.2.25:50010
2011-11-01 22:39:53,336 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-7734816649100892246_235458 src: /10.1.2.26:36320 dest: /10.1.2.25:50010
2011-11-01 22:39:57,162 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.1.2.22:37400, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
cliID: DFSClient_-858857344, offset: 0, srvID:
DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
blk_-1542408817127580486_235458, duration: 21156605250
2011-11-01 22:39:57,162 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
block blk_-1542408817127580486_235458 terminating

Thanks
Weihua

Re: Data node problem after reinstall

Posted by Jean-Daniel Cryans <jd...@apache.org>.
I don't see anything wrong in that log, do you actually have WARN or
ERROR level log lines? Those might be a better start.

Also please explain how you figured that the DN can only accept
replicas and nothing from HBase, hopefully with evidence. This will
greatly help.

Thx,

J-D

On Tue, Nov 1, 2011 at 12:28 AM, Weihua JIANG <we...@gmail.com> wrote:
> Hi all,
>
> I am not sure whether it is a hbase problem or a hdfs problem.
>
> When have 8 datanodes & regionservers each with 12 disks. One data node is
> down due to its system disk broken. After replacing the disk and reinstall
> the OS, we tried to online this DN & region server. The region server is
> OK. But, the data node seems can only accept block replication from other
> data nodes, but failed to accept writing from HBase. Even after we removed
> all content of data dir and restarted the DN, this problem still exists.
> The log is:
> 2011-11-01 22:39:44,397 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_-6413273237261014407_77967 src: /10.1.2.23:56859 dest:
> /10.1.2.25:50010of size 67108864
> 2011-11-01 22:39:45,959 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_-6561707380479467846_78302 src: /10.1.2.20:47578 dest:
> /10.1.2.25:50010of size 67108864
> 2011-11-01 22:39:46,314 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-6233096255264073662_235292 src: /10.1.2.26:36311 dest: /10.1.2.25:50010
> 2011-11-01 22:39:46,829 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.1.2.21:35450, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
> cliID: DFSClient_1189047403, offset: 0, srvID:
> DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
> blk_7603159785320952833_235458, duration: 20790059066
> 2011-11-01 22:39:46,829 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
> block blk_7603159785320952833_235458 terminating
> 2011-11-01 22:39:46,850 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_7980572561428708688_235458 src: /10.1.2.19:37630 dest: /10.1.2.25:50010
> 2011-11-01 22:39:47,093 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.1.2.19:37611, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
> cliID: DFSClient_-81251145, offset: 0, srvID:
> DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
> blk_-9067419333696936066_235458, duration: 20478973800
> 2011-11-01 22:39:47,093 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
> block blk_-9067419333696936066_235458 terminating
> 2011-11-01 22:39:47,117 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-495608789656244385_235458 src: /10.1.2.21:35468 dest: /10.1.2.25:50010
> 2011-11-01 22:39:47,185 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.1.2.23:56845, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
> cliID: DFSClient_1647768037, offset: 0, srvID:
> DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
> blk_-7935479347615843321_235458, duration: 20595122675
> 2011-11-01 22:39:47,185 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
> block blk_-7935479347615843321_235458 terminating
> 2011-11-01 22:39:47,325 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_-6233096255264073662_235292 src: /10.1.2.26:36311 dest:
> /10.1.2.25:50010of size 67108864
> 2011-11-01 22:39:49,981 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.1.2.25:60888, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
> cliID: DFSClient_-1345645880, offset: 0, srvID:
> DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
> blk_2614344148027463064_235458, duration: 19952693395
> 2011-11-01 22:39:49,982 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for
> block blk_2614344148027463064_235458 terminating
> 2011-11-01 22:39:49,999 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_5157459941530714338_235458 src: /10.1.2.25:60892 dest: /10.1.2.25:50010
> 2011-11-01 22:39:51,258 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-6108054911705086653_39962 src: /10.1.2.21:35475 dest: /10.1.2.25:50010
> 2011-11-01 22:39:51,259 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_-6108054911705086653_39962 src: /10.1.2.21:35475 dest:
> /10.1.2.25:50010of size 124
> 2011-11-01 22:39:52,357 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-5980257649446708968_78270 src: /10.1.2.23:56867 dest: /10.1.2.25:50010
> 2011-11-01 22:39:53,336 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-7734816649100892246_235458 src: /10.1.2.26:36320 dest: /10.1.2.25:50010
> 2011-11-01 22:39:57,162 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.1.2.22:37400, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE,
> cliID: DFSClient_-858857344, offset: 0, srvID:
> DS-1739614687-10.1.2.25-50010-1320156634502, blockid:
> blk_-1542408817127580486_235458, duration: 21156605250
> 2011-11-01 22:39:57,162 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
> block blk_-1542408817127580486_235458 terminating
>
> Thanks
> Weihua
>