[jira] [Commented] (HDFS-4949) Centralized cache management in HDFS

["Sanjay Radia (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707288#comment-13707288 ] 

Sanjay Radia commented on HDFS-4949:
------------------------------------

I think we can treat the RAM copies as replicas - this fits into the generalized tiered-storage architecture as described in HDFS-2832 (Ram, flash, disk.)
* Block reports will indicate the storage type.
* NN will store the storage type in the Block map
* Block locations returned by NN to client will have storage type (i.e. don't need the IsCached flag).
** NN will order the replicas locations based on closeness and speed; this will mean that the client side will automatically go to the best place (although we can have a smarter client do something different if desired.)
* NN will not count Ram replicas towards the normal replica count - this is one area where the ram replicas are treated differently. 
* This can support a usage model where the ram replicas are at each or only some of the disk replica locations.  
                
> Centralized cache management in HDFS
> ------------------------------------
>
>                 Key: HDFS-4949
>                 URL: https://issues.apache.org/jira/browse/HDFS-4949
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode, namenode
>    Affects Versions: 3.0.0, 2.2.0
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>         Attachments: caching-design-doc-2013-07-02.pdf
>
>
> HDFS currently has no support for managing or exposing in-memory caches at datanodes. This makes it harder for higher level application frameworks like Hive, Pig, and Impala to effectively use cluster memory, because they cannot explicitly cache important datasets or place their tasks for memory locality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira