You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Alexey Goncharuk (JIRA)" <ji...@apache.org> on 2015/07/20 23:14:05 UTC

[jira] [Created] (SPARK-9203) Make filesystem pluggable in BlockStore and BlockManager

Alexey Goncharuk created SPARK-9203:
---------------------------------------

             Summary: Make filesystem pluggable in BlockStore and BlockManager
                 Key: SPARK-9203
                 URL: https://issues.apache.org/jira/browse/SPARK-9203
             Project: Spark
          Issue Type: Improvement
            Reporter: Alexey Goncharuk


I was looking through the code in order to understand better how RDD is persisted to Tachyon off-heap filesystem. It looks like that the Tachyon filesystem is hard-coded and there is no way to switch to another in-memory filesystem. I think it would be great if the implementation of the BlockManager and BlockStore would be able to plug in another filesystem.

For example, Apache Ignite also has an implementation of in-memory filesystem which can store data in on-heap and off-heap formats. It would be great if it could integrate with Spark.

Apache Ignite In-Memory Filesystem: https://ignite.incubator.apache.org/features/igfs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org