[jira] Updated: (HADOOP-4663) Datanode should delete files under tmp when upgraded from 0.17

["dhruba borthakur (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-4663:
-------------------------------------

    Attachment: deleteTmp2.patch

When a datanode receives a replication request, it starts the block in a separate directory. These blocks will be cleaned up when the datanode restarts. The general algorithm is as follows:

1. When a datanode receives a block from a client, it starts the block in datadir/tmp directory.
2. When a datanode receives a block as part of a replication request, it starts the block in datadir//tmp_replication directory
3. When a block is finalised, it moves from any of the above two temporary directories to its final location.
4. When a datanode restarts, it moves all the blocks from datadir/tmp directory to their final location. These blocsk were part of writes by clients and the client-namenode-genstamp protocol will ensure that the most-uptodate replicas of the block will be maintained.
4. When a datanode restarts, the blocks in datadir/tmp_replication directory are removed. No recovery is needed for these blocks.

The question that remains is what do we need to do for blocks that are moved as part of rebalancer? I think the blocks that rebalancer moves should go into tmp_replication as well. 


> Datanode should delete files under tmp when upgraded from 0.17
> --------------------------------------------------------------
>
>                 Key: HADOOP-4663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4663
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.18.0
>            Reporter: Raghu Angadi
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.18.3
>
>         Attachments: deleteTmp.patch, deleteTmp2.patch
>
>
> Before 0.18, when Datanode restarts, it deletes files under data-dir/tmp  directory since these files are not valid anymore. But in 0.18 it moves these files to normal directory incorrectly making them valid blocks. One of the following would work :
> - remove the tmp files during upgrade, or
> - if the files under /tmp are in pre-18 format (i.e. no generation), delete them.
> Currently effect of this bug is that, these files end up failing block verification and eventually get deleted. But cause incorrect over-replication at the namenode before that.
> Also it looks like our policy regd treating files under tmp needs to be defined better. Right now there are probably one or two more bugs with it. Dhruba, please file them if you rememeber.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.