[jira] [Comment Edited] (SPARK-6112) Provide external block store support through HDFS RAM_DISK

["Arpit Agarwal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606551#comment-14606551 ] 

Arpit Agarwal edited comment on SPARK-6112 at 6/29/15 10:31 PM:
----------------------------------------------------------------

Hi [~bghit], thanks for the repro steps. I tried it out and saw the output splits from {{saveAsTextFile}} go to RAM_DISK.

I used Spark v 1.4.0 + Apache Hadoop 2.6.0. Let me try it out with 2.7.0 also since that's what you had.


was (Author: arpitagarwal):
Hi [~bghit], thanks for t

> Provide external block store support through HDFS RAM_DISK
> ----------------------------------------------------------
>
>                 Key: SPARK-6112
>                 URL: https://issues.apache.org/jira/browse/SPARK-6112
>             Project: Spark
>          Issue Type: New Feature
>          Components: Block Manager
>            Reporter: Zhan Zhang
>         Attachments: SparkOffheapsupportbyHDFS.pdf
>
>
> HDFS Lazy_Persist policy provide possibility to cache the RDD off_heap in hdfs. We may want to provide similar capacity to Tachyon by leveraging hdfs RAM_DISK feature, if the user environment does not have tachyon deployed. 
> With this feature, it potentially provides possibility to share RDD in memory across different jobs and even share with jobs other than spark, and avoid the RDD recomputation if executors crash. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org