You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2007/04/07 00:03:32 UTC

[jira] Created: (HADOOP-1220) block not found in blockMap

block not found in blockMap
---------------------------

                 Key: HADOOP-1220
                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
             Project: Hadoop
          Issue Type: Bug
          Components: dfs
            Reporter: Koji Noguchi


>From Hadoop-973  debug message, we had datanode constantly printing out the following message.

...
2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
  at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
  at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
  at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
  at java.lang.Thread.run(Thread.java:619)

2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464 to /72.30.127.164
2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
  at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
  at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
  at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
  at java.lang.Thread.run(Thread.java:619)
....

 Ten minutes before these logs, there was 
2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031

There was a  file for that block in the directory.
/../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HADOOP-1220) block not found in blockMap

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur reassigned HADOOP-1220:
----------------------------------------

    Assignee: dhruba borthakur

> block not found in blockMap
> ---------------------------
>
>                 Key: HADOOP-1220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Koji Noguchi
>         Assigned To: dhruba borthakur
>         Attachments: datanodeDeleteBlocks.patch
>
>
> From Hadoop-973  debug message, we had datanode constantly printing out the following message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464 to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> ....
>  Ten minutes before these logs, there was 
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031
> There was a  file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1220) block not found in blockMap

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491727 ] 

dhruba borthakur commented on HADOOP-1220:
------------------------------------------

This problem occured again. This time the block in question does not appear in the dfs data directory. The namenode is requesting a block to be deleted. The datanode tries this operation and encounters an error because the block is not in the blockMap. The processCommand() method raises an exception. The code is such that the variable lastBlockReport is not set if processCommand() raises an exception. This means that the datanode immediately send another block report to the namenode.

In short, the above condition causes the datanode to send blockReports almost once every second!

I propose that we do the following:

1. in Datanode.offerService, replace the following piece of code 

          DatanodeCommand cmd = namenode.blockReport(dnRegistration,
                                                     data.getBlockReport());
          processCommand(cmd);
          lastBlockReport = now;


with

          DatanodeCommand cmd = namenode.blockReport(dnRegistration,
                                                     data.getBlockReport());
          lastBlockReport = now;
          processCommand(cmd);
          
2. In FSDataSet.invalidate:
    a) continue to process all blocks in invalidBlks[] even if one in the middle encounters a problem.
    b) if getFile() returns null, still invoke volumeMap.get() and print whether we found the block in 
       volumes or not. The volumeMap is used to generate the blockReport and this might help in debugging.







> block not found in blockMap
> ---------------------------
>
>                 Key: HADOOP-1220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Koji Noguchi
>
> From Hadoop-973  debug message, we had datanode constantly printing out the following message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464 to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> ....
>  Ten minutes before these logs, there was 
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031
> There was a  file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1220) block not found in blockMap

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491775 ] 

Raghu Angadi commented on HADOOP-1220:
--------------------------------------


It at least confirms to us if f.delete() is indeed the problem or not. f.exists() could be included temporarily until we find where the bug is.


> block not found in blockMap
> ---------------------------
>
>                 Key: HADOOP-1220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Koji Noguchi
>         Assigned To: dhruba borthakur
>         Attachments: datanodeDeleteBlocks.patch
>
>
> From Hadoop-973  debug message, we had datanode constantly printing out the following message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464 to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> ....
>  Ten minutes before these logs, there was 
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031
> There was a  file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1220) block not found in blockMap

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491766 ] 

Raghu Angadi commented on HADOOP-1220:
--------------------------------------

> The failure to delete the fail seems [...]
should be: .. delete the FILE ...


> block not found in blockMap
> ---------------------------
>
>                 Key: HADOOP-1220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Koji Noguchi
>         Assigned To: dhruba borthakur
>         Attachments: datanodeDeleteBlocks.patch
>
>
> From Hadoop-973  debug message, we had datanode constantly printing out the following message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464 to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> ....
>  Ten minutes before these logs, there was 
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031
> There was a  file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1220) block not found in blockMap

Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491761 ] 

Koji Noguchi commented on HADOOP-1220:
--------------------------------------

bq. This time the block in question does not appear in the dfs data directory.

We've seen both cases.

1) block exist on local dfs dir. 
     Error messages repeats every second.

2) block doesn't exist on local dfs dir.
     Error message only shows up once.


It happened when a large number of files were deleted on the cluster.


> block not found in blockMap
> ---------------------------
>
>                 Key: HADOOP-1220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Koji Noguchi
>
> From Hadoop-973  debug message, we had datanode constantly printing out the following message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464 to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> ....
>  Ten minutes before these logs, there was 
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031
> There was a  file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1220) block not found in blockMap

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1220:
-------------------------------------

    Attachment: datanodeDeleteBlocks.patch

Implemented the changes described in the previous comment.

> block not found in blockMap
> ---------------------------
>
>                 Key: HADOOP-1220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Koji Noguchi
>         Attachments: datanodeDeleteBlocks.patch
>
>
> From Hadoop-973  debug message, we had datanode constantly printing out the following message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464 to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> ....
>  Ten minutes before these logs, there was 
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031
> There was a  file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1220) block not found in blockMap

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491765 ] 

Raghu Angadi commented on HADOOP-1220:
--------------------------------------


The failure to delete the fail seems to happen when a lot of blocks are deleted at once. Should we trust f.delete() when it returns true or should we double check?



> block not found in blockMap
> ---------------------------
>
>                 Key: HADOOP-1220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Koji Noguchi
>         Assigned To: dhruba borthakur
>         Attachments: datanodeDeleteBlocks.patch
>
>
> From Hadoop-973  debug message, we had datanode constantly printing out the following message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464 to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> ....
>  Ten minutes before these logs, there was 
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031
> There was a  file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1220) block not found in blockMap

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487366 ] 

Raghu Angadi commented on HADOOP-1220:
--------------------------------------

Since we have "Deleting ..." log message, it implies that Java "f.delete(file)" returned true. But looks s like the file is actually not deleted. 

> block not found in blockMap
> ---------------------------
>
>                 Key: HADOOP-1220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Koji Noguchi
>
> From Hadoop-973  debug message, we had datanode constantly printing out the following message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464 to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> ....
>  Ten minutes before these logs, there was 
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031
> There was a  file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1220) block not found in blockMap

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491772 ] 

dhruba borthakur commented on HADOOP-1220:
------------------------------------------

I am not too inclined to put in a check to f.exists() after every call to f.delete(). If you are doubting the correctness of f.delete() then what guarantees does f.exists() have? But I am printing the full path of the file when a file gets deleted, earlier it was only printing the blockid. This message might help in determining if we are deleting the correct file or not.

> block not found in blockMap
> ---------------------------
>
>                 Key: HADOOP-1220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Koji Noguchi
>         Assigned To: dhruba borthakur
>         Attachments: datanodeDeleteBlocks.patch
>
>
> From Hadoop-973  debug message, we had datanode constantly printing out the following message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464 to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> ....
>  Ten minutes before these logs, there was 
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031
> There was a  file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.