You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Logan Hardy <lo...@gmail.com> on 2013/09/06 17:42:17 UTC

DataNode: Can't replicate block

I'm seeing thousands of the following messages per day on my Datanodes. In
every single message the NameNode recorded length is exactly
"9223372036854775807". That's 8192 Petabytes. Ideas? I posted this error as
part of a larger question that went's unanswered on the CDH user group but
I really want to understand this specific message so I'm breaking it out
here.

hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0

2013-09-05 03:57:56,765 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_-917008810045374143_2222520
because on-disk length 843090 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:57:59,765 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_4638855469361221841_2222522
because on-disk length 108990 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:57:59,766 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_-3096141510869821776_2222526
because on-disk length 706015 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:58:02,767 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_1061908974542106736_2222528
because on-disk length 659123 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:58:08,772 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_235372784090791353_2222585
because on-disk length 4609178 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:59:17,805 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_8980835874496757824_2223043
because on-disk length 166263 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:59:20,807 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_9066832331074569403_2223100
because on-disk length 33742 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:59:20,808 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_-3987096455745701841_2223102
because on-disk length 4609178 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:59:56,831 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_3246916891446829945_2223298
because on-disk length 108990 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:59:56,832 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_-8519971491515089660_2223300
because on-disk length 697887 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:59:59,833 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_7166663523039348005_2223308
because on-disk length 241259 is shorter than NameNode recorded length
9223372036854775807
2013-09-05 03:59:59,834 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
BP-1353222332-10.45.100.25-1360087637866:blk_3360556618070774912_2223310
because on-disk length 166263 is shorter than NameNode recorded length
9223372036854775807

- Logan-

Re: DataNode: Can't replicate block

Posted by Andre Kelpe <ak...@concurrentinc.com>.
It looks like an overflow somewhere 9223372036854775807 ==
0x7fffffffffffffffL == java.lang.Long.MAX_VALUE.

- André

On Fri, Sep 6, 2013 at 5:42 PM, Logan Hardy <lo...@gmail.com> wrote:
> I'm seeing thousands of the following messages per day on my Datanodes. In
> every single message the NameNode recorded length is exactly
> "9223372036854775807". That's 8192 Petabytes. Ideas? I posted this error as
> part of a larger question that went's unanswered on the CDH user group but I
> really want to understand this specific message so I'm breaking it out here.
>
> hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0
>
> 2013-09-05 03:57:56,765 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-917008810045374143_2222520
> because on-disk length 843090 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:57:59,765 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_4638855469361221841_2222522
> because on-disk length 108990 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:57:59,766 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-3096141510869821776_2222526
> because on-disk length 706015 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:58:02,767 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_1061908974542106736_2222528
> because on-disk length 659123 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:58:08,772 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_235372784090791353_2222585
> because on-disk length 4609178 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:17,805 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_8980835874496757824_2223043
> because on-disk length 166263 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:20,807 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_9066832331074569403_2223100
> because on-disk length 33742 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:20,808 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-3987096455745701841_2223102
> because on-disk length 4609178 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:56,831 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_3246916891446829945_2223298
> because on-disk length 108990 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:56,832 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-8519971491515089660_2223300
> because on-disk length 697887 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:59,833 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_7166663523039348005_2223308
> because on-disk length 241259 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:59,834 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_3360556618070774912_2223310
> because on-disk length 166263 is shorter than NameNode recorded length
> 9223372036854775807
>
> - Logan-



-- 
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com

Re: DataNode: Can't replicate block

Posted by Andre Kelpe <ak...@concurrentinc.com>.
It looks like an overflow somewhere 9223372036854775807 ==
0x7fffffffffffffffL == java.lang.Long.MAX_VALUE.

- André

On Fri, Sep 6, 2013 at 5:42 PM, Logan Hardy <lo...@gmail.com> wrote:
> I'm seeing thousands of the following messages per day on my Datanodes. In
> every single message the NameNode recorded length is exactly
> "9223372036854775807". That's 8192 Petabytes. Ideas? I posted this error as
> part of a larger question that went's unanswered on the CDH user group but I
> really want to understand this specific message so I'm breaking it out here.
>
> hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0
>
> 2013-09-05 03:57:56,765 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-917008810045374143_2222520
> because on-disk length 843090 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:57:59,765 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_4638855469361221841_2222522
> because on-disk length 108990 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:57:59,766 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-3096141510869821776_2222526
> because on-disk length 706015 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:58:02,767 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_1061908974542106736_2222528
> because on-disk length 659123 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:58:08,772 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_235372784090791353_2222585
> because on-disk length 4609178 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:17,805 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_8980835874496757824_2223043
> because on-disk length 166263 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:20,807 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_9066832331074569403_2223100
> because on-disk length 33742 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:20,808 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-3987096455745701841_2223102
> because on-disk length 4609178 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:56,831 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_3246916891446829945_2223298
> because on-disk length 108990 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:56,832 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-8519971491515089660_2223300
> because on-disk length 697887 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:59,833 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_7166663523039348005_2223308
> because on-disk length 241259 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:59,834 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_3360556618070774912_2223310
> because on-disk length 166263 is shorter than NameNode recorded length
> 9223372036854775807
>
> - Logan-



-- 
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com

Re: DataNode: Can't replicate block

Posted by Andre Kelpe <ak...@concurrentinc.com>.
It looks like an overflow somewhere 9223372036854775807 ==
0x7fffffffffffffffL == java.lang.Long.MAX_VALUE.

- André

On Fri, Sep 6, 2013 at 5:42 PM, Logan Hardy <lo...@gmail.com> wrote:
> I'm seeing thousands of the following messages per day on my Datanodes. In
> every single message the NameNode recorded length is exactly
> "9223372036854775807". That's 8192 Petabytes. Ideas? I posted this error as
> part of a larger question that went's unanswered on the CDH user group but I
> really want to understand this specific message so I'm breaking it out here.
>
> hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0
>
> 2013-09-05 03:57:56,765 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-917008810045374143_2222520
> because on-disk length 843090 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:57:59,765 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_4638855469361221841_2222522
> because on-disk length 108990 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:57:59,766 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-3096141510869821776_2222526
> because on-disk length 706015 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:58:02,767 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_1061908974542106736_2222528
> because on-disk length 659123 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:58:08,772 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_235372784090791353_2222585
> because on-disk length 4609178 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:17,805 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_8980835874496757824_2223043
> because on-disk length 166263 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:20,807 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_9066832331074569403_2223100
> because on-disk length 33742 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:20,808 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-3987096455745701841_2223102
> because on-disk length 4609178 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:56,831 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_3246916891446829945_2223298
> because on-disk length 108990 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:56,832 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-8519971491515089660_2223300
> because on-disk length 697887 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:59,833 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_7166663523039348005_2223308
> because on-disk length 241259 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:59,834 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_3360556618070774912_2223310
> because on-disk length 166263 is shorter than NameNode recorded length
> 9223372036854775807
>
> - Logan-



-- 
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com

Re: DataNode: Can't replicate block

Posted by Andre Kelpe <ak...@concurrentinc.com>.
It looks like an overflow somewhere 9223372036854775807 ==
0x7fffffffffffffffL == java.lang.Long.MAX_VALUE.

- André

On Fri, Sep 6, 2013 at 5:42 PM, Logan Hardy <lo...@gmail.com> wrote:
> I'm seeing thousands of the following messages per day on my Datanodes. In
> every single message the NameNode recorded length is exactly
> "9223372036854775807". That's 8192 Petabytes. Ideas? I posted this error as
> part of a larger question that went's unanswered on the CDH user group but I
> really want to understand this specific message so I'm breaking it out here.
>
> hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0
>
> 2013-09-05 03:57:56,765 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-917008810045374143_2222520
> because on-disk length 843090 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:57:59,765 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_4638855469361221841_2222522
> because on-disk length 108990 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:57:59,766 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-3096141510869821776_2222526
> because on-disk length 706015 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:58:02,767 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_1061908974542106736_2222528
> because on-disk length 659123 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:58:08,772 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_235372784090791353_2222585
> because on-disk length 4609178 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:17,805 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_8980835874496757824_2223043
> because on-disk length 166263 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:20,807 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_9066832331074569403_2223100
> because on-disk length 33742 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:20,808 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-3987096455745701841_2223102
> because on-disk length 4609178 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:56,831 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_3246916891446829945_2223298
> because on-disk length 108990 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:56,832 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_-8519971491515089660_2223300
> because on-disk length 697887 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:59,833 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_7166663523039348005_2223308
> because on-disk length 241259 is shorter than NameNode recorded length
> 9223372036854775807
> 2013-09-05 03:59:59,834 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block
> BP-1353222332-10.45.100.25-1360087637866:blk_3360556618070774912_2223310
> because on-disk length 166263 is shorter than NameNode recorded length
> 9223372036854775807
>
> - Logan-



-- 
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com