You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Rajit (JIRA)" <ji...@apache.org> on 2011/09/23 02:21:26 UTC
[jira] [Created] (HDFS-2359) NPE found in Namenode log while Disk
failed during distcp
NPE found in Namenode log while Disk failed during distcp
----------------------------------------------------------
Key: HDFS-2359
URL: https://issues.apache.org/jira/browse/HDFS-2359
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Reporter: Rajit
Scenario:
I have a cluster of 4 DN ,each of them have 12disks.
In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
In one thread
$hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
In another thread in a datanode
$ chmod 000 /grid/{0,1,2,3}/hadoop/var/hdfs/data
Log Snippet from the Datanode
=============
2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
java.io.IOException: Error in deleting blocks.
at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
at java.lang.Thread.run(Thread.java:619)
2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
ipcPort=8020):DataXceiver
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
at java.lang.Thread.run(Thread.java:619)
2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114966#comment-13114966 ]
Jonathan Eagles commented on HDFS-2359:
---------------------------------------
Thanks, Suresh!
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles updated HDFS-2359:
----------------------------------
Attachment: HDFS-2359-branch-0.20-security.patch
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Rajit (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajit updated HDFS-2359:
------------------------
Summary: NPE found in Datanode log while Disk failed during different HDFS operation (was: NPE found in Namenode log while Disk failed during distcp )
Thanks Aaron, its Datanode , my bad
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Rajit
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2359) NPE found in Namenode log while Disk
failed during distcp
Posted by "Rajit (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajit updated HDFS-2359:
------------------------
Description:
Scenario:
I have a cluster of 4 DN ,each of them have 12disks.
In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
In one thread
$hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
In another thread in a datanode
$ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
Log Snippet from the Datanode
=============
2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
java.io.IOException: Error in deleting blocks.
at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
at java.lang.Thread.run(Thread.java:619)
2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
ipcPort=8020):DataXceiver
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
at java.lang.Thread.run(Thread.java:619)
2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
was:
Scenario:
I have a cluster of 4 DN ,each of them have 12disks.
In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
In one thread
$hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
In another thread in a datanode
$ chmod 000 /grid/{0,1,2,3}/hadoop/var/hdfs/data
Log Snippet from the Datanode
=============
2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
java.io.IOException: Error in deleting blocks.
at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
at java.lang.Thread.run(Thread.java:619)
2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
ipcPort=8020):DataXceiver
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
at java.lang.Thread.run(Thread.java:619)
2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
> NPE found in Namenode log while Disk failed during distcp
> ----------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Rajit
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114817#comment-13114817 ]
Hadoop QA commented on HDFS-2359:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496504/HDFS-2359-branch-0.20-security.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1290//console
This message is automatically generated.
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114901#comment-13114901 ]
Suresh Srinivas commented on HDFS-2359:
---------------------------------------
I have committed this patch. Thank you Jonathan for the patch.
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114805#comment-13114805 ]
Jonathan Eagles commented on HDFS-2359:
---------------------------------------
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114779#comment-13114779 ]
Jonathan Eagles commented on HDFS-2359:
---------------------------------------
This NPE was fixed in trunk as part of HDFS-1655. Cherry picking the NPE fix to create the patch for this JIRA.
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Namenode log while Disk
failed during distcp
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113057#comment-13113057 ]
Aaron T. Myers commented on HDFS-2359:
--------------------------------------
Looks like the description of this JIRA should say "NameNode", not "DataNode", right?
> NPE found in Namenode log while Disk failed during distcp
> ----------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Rajit
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles reassigned HDFS-2359:
-------------------------------------
Assignee: Jonathan Eagles
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Suresh Srinivas (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-2359:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Fix For: 0.20.205.0
>
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Suresh Srinivas (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-2359:
----------------------------------
Fix Version/s: 0.20.205.0
Hadoop Flags: Reviewed
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Fix For: 0.20.205.0
>
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113187#comment-13113187 ]
Suresh Srinivas commented on HDFS-2359:
---------------------------------------
Rajit, what release is this happening. Could you please set the Affects Version/s with release where you found this problem.
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Rajit
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114807#comment-13114807 ]
Jonathan Eagles commented on HDFS-2359:
---------------------------------------
See above comment on why no tests were provided. This patch should go into branch-0.20-security and branch-0.20-security-205.
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles updated HDFS-2359:
----------------------------------
Status: Patch Available (was: Open)
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Rajit (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113132#comment-13113132 ]
Rajit commented on HDFS-2359:
-----------------------------
Also I have seen , this NullPointer exception can happen any of the scenarios,
like dfs -copyFromLocal/-put/-cp/-rm or distcp. Its not definitive. I have seen
this in many times in different occasions. But everytime it happens when I
try to simulate Disk Fail by changing permission of Data dir as 000
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Rajit
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114789#comment-13114789 ]
Jonathan Eagles commented on HDFS-2359:
---------------------------------------
No tests were added due to the difficulty of testing a private method of a static inner class.
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114895#comment-13114895 ]
Suresh Srinivas commented on HDFS-2359:
---------------------------------------
+1 for the patch.
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
> Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk
failed during different HDFS operation
Posted by "Rajit (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajit updated HDFS-2359:
------------------------
Affects Version/s: 0.20.205.0
Suresh, thanks for correcting. I am seeing this in .20.205 unreleased version
> NPE found in Datanode log while Disk failed during different HDFS operation
> ---------------------------------------------------------------------------
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3"
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one Datanode, by making Data Directory permission 000, The distcp job is successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =============
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:xxxx, storageID=xx-xxxxxxxxxxxx-xx.xxx.xxx.xxx-xxxx-xxxxxxxxxxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira