[jira] Commented: (HADOOP-4058) Transparent archival and restore of files from HDFS

["dhruba borthakur (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627909#action_12627909 ] 

dhruba borthakur commented on HADOOP-4058:
------------------------------------------

Hadoop user's have been using the Hadoop clusters as a queryable archive warehouse. This means that data that once gets into the warehouse is very unlikely to be deleted. This puts tremendous pressure on adding additional storage capacity to the production cluster.

There could be a set of storage-heavy nodes that cannot be added to the production cluster because do not have enough memory and CPU. One option would be to use this old-cluster to archive old files from the production cluster.

A layer of software can scan the file system in the production cluster to find files with the earliest access times (HADOOP-1869). These files can be moved to the old-cluster and the original file in the production cluster can be replaced by a symbolic link (via HADOOP-4044). An access to read the original file still works because of the symbolic link. Some other piece of software periodically scans the old-cluster, finds out files that were accessed recently, and tries to move them back to the production cluster.

The advantage of this approach is that it is "layered"... it is not built into HDFS but depends on two artifacts of HDFS: symbolic links and access-times. I hate to put more and more intelligence into core-hdfs, otherwise the code becomes very bloated and difficult to maintain.


> Transparent archival and restore of files from HDFS
> ---------------------------------------------------
>
>                 Key: HADOOP-4058
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4058
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> There should be a facility to migrate old files away from a production cluster. Access to those files from applications should continue to work transparently, without changing application code, but maybe with reduced performance. The policy engine  that does this could be layered on HDFS rather than being built into HDFS itself.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.