You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/04/06 06:02:00 UTC
[jira] [Commented] (SPARK-10399) Off Heap Memory Access for non-JVM
libraries (C++)
[ https://issues.apache.org/jira/browse/SPARK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428000#comment-16428000 ]
Apache Spark commented on SPARK-10399:
--------------------------------------
User 'kiszk' has created a pull request for this issue:
https://github.com/apache/spark/pull/20991
> Off Heap Memory Access for non-JVM libraries (C++)
> --------------------------------------------------
>
> Key: SPARK-10399
> URL: https://issues.apache.org/jira/browse/SPARK-10399
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Paul Weiss
> Priority: Major
>
> *Summary*
> Provide direct off-heap memory access to an external non-JVM program such as a c++ library within the Spark running JVM/executor. As Spark moves to storing all data into off heap memory it makes sense to provide access points to the memory for non-JVM programs.
> ----
> *Assumptions*
> * Zero copies will be made during the call into non-JVM library
> * Access into non-JVM libraries will be accomplished via JNI
> * A generic JNI interface will be created so that developers will not need to deal with the raw JNI call
> * C++ will be the initial target non-JVM use case
> * memory management will remain on the JVM/Spark side
> * the API from C++ will be similar to dataframes as much as feasible and NOT require expert knowledge of JNI
> * Data organization and layout will support complex (multi-type, nested, etc.) types
> ----
> *Design*
> * Initially Spark JVM -> non-JVM will be supported
> * Creating an embedded JVM with Spark running from a non-JVM program is initially out of scope
> ----
> *Technical*
> * GetDirectBufferAddress is the JNI call used to access byte buffer without copy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org