[jira] [Commented] (SPARK-1957) Pluggable disk store for BlockManager

["Raymond Liu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012081#comment-14012081 ] 

Raymond Liu commented on SPARK-1957:
------------------------------------

Initial pull request at : https://github.com/apache/spark/pull/907

The current approaching as below : 

A diskStoreManager layer is added to manage multiple layered disk stores, each layer of store employ a quota on volume. except the last store, which should be the fall back media and thus do not have hard quota limitation. On each Store layer, when the quota is reached, the further data is written to lower layer. The diskStoreManager is the common interface that BlockManager call into.

The shuffle realted staffs take the assumption that disk store is a pure file based store, and take short cut path to write file directly, and also has no easy way to control quota, especially when shuffle consolidate is involved. So in the solution, the shuffle related code is pick out, and all shuffle data goto the last fallback data store. say most probably HDDs.

Should have shuffle related code not access file directly, and working with current several proposals on shuffle framework to make it fit in pluggable store infrastructure in the future.


> Pluggable disk store for BlockManager
> -------------------------------------
>
>                 Key: SPARK-1957
>                 URL: https://issues.apache.org/jira/browse/SPARK-1957
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>            Reporter: Raymond Liu
>
> As the first step toward the goal of SPAK-1733, support a pluggable disk store to allow different disk storage to be plug into the BlockManager's DiskStore layer.



--
This message was sent by Atlassian JIRA
(v6.2#6252)