You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dmitriy Setrakyan (JIRA)" <ji...@apache.org> on 2015/05/09 19:58:00 UTC
[jira] [Comment Edited] (SPARK-6479) Create external block store
API
[ https://issues.apache.org/jira/browse/SPARK-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536793#comment-14536793 ]
Dmitriy Setrakyan edited comment on SPARK-6479 at 5/9/15 5:57 PM:
------------------------------------------------------------------
I believe that you maybe able to leverage Apache Ignite off-heap memory implementation for Spark external block store implementation.
Apache Ignite already comes with a very easy to use off-heap store based on JCache (JSR107) API. It also can be used as a distributed partitioned off-heap cache, if needed.
More information can be found here:
http://apacheignite.readme.io/v1.0/docs/off-heap-memory
You can interact with Ignite just like with a {{ConcurrentHashMap}}. If off-heap mode is configured, all keys and values will be stored outside of JVM heap, in the off-heap memory.
{code}
cache.put("Hello", 1);
val = cache.get("Hello");
oldVal = cache.getAndPutIfAbsent("Hello", 11);
success = cache.putIfAbsent("World", 22);
oldVal = cache.getAndReplace("Hello", 11);
success = cache.replace("World", 22);
success = cache.replace("World", 2, 22);
success = cache.remove("Hello", 1);
{code}
was (Author: dsetrakyan):
I believe that you maybe able to leverage Apache Ignite off-heap memory implementation for Spark external block store implementation.
Apache Ignite already comes with a very easy to use off-heap store based on JCache (JSR107) API. It also can be used as a distributed partitioned off-heap cache, if needed.
More information can be found here:
http://apacheignite.readme.io/v1.0/docs/off-heap-memory
You can interact with Ignite just like with a {{ConcurrentHashMap}}. If off-heap mode is configured, all keys and values will be stored outside of JVM heap, in the off-heap memory.
{code}
cache.put("Hello", 1);
Integer val = cache.get("Hello");
Integer oldVal = cache.getAndPutIfAbsent("Hello", 11);
boolean success = cache.putIfAbsent("World", 22);
oldVal = cache.getAndReplace("Hello", 11);
success = cache.replace("World", 22);
success = cache.replace("World", 2, 22);
success = cache.remove("Hello", 1);
{code}
> Create external block store API
> -------------------------------
>
> Key: SPARK-6479
> URL: https://issues.apache.org/jira/browse/SPARK-6479
> Project: Spark
> Issue Type: Improvement
> Components: Block Manager, Spark Core
> Reporter: Reynold Xin
> Assignee: Zhan Zhang
> Fix For: 1.4.0
>
> Attachments: SPARK-6479.pdf, SPARK-6479OffheapAPIdesign (1).pdf, SPARK-6479OffheapAPIdesign.pdf, SparkOffheapsupportbyHDFS.pdf, spark-6479-tachyon.patch
>
>
> Would be great to create APIs for external block stores, rather than doing a bunch of if statements everywhere.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org