You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2019/06/19 02:36:00 UTC
[jira] [Assigned] (SPARK-28102) Add configuration for selecting LZ4
implementation (safe, unsafe, JNI)
[ https://issues.apache.org/jira/browse/SPARK-28102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-28102:
------------------------------------
Assignee: Apache Spark (was: Josh Rosen)
> Add configuration for selecting LZ4 implementation (safe, unsafe, JNI)
> ----------------------------------------------------------------------
>
> Key: SPARK-28102
> URL: https://issues.apache.org/jira/browse/SPARK-28102
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Josh Rosen
> Assignee: Apache Spark
> Priority: Major
>
> Spark's use of {{lz4-java}} ends up calling {{Lz4Factory.fastestInstance}}, which attempts to load JNI libraries and falls back on Java implementations in case the JNI library cannot be loaded or initialized.
> I run Spark in a configuration where the JNI libraries don't work, so I'd like to configure LZ4 to not even attempt to use JNI code: if the JNI library loads but cannot be initialized then the fallback code path involves catching an exception and this is slow because the exception is thrown under a static initializer lock (leading to significant lock contention because the filling of stacktraces is done while holding this lock).
> I propose to introduce a {{spark.io.compression.lz4.factory}} configuration for selecting the LZ4 implementation, allowing users to disable the use of the JNI library without having to recompile Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org