You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2019/06/19 15:37:01 UTC
[jira] [Updated] (SPARK-28102) Failed LZ4 JNI initialization is
repeatedly re-attempted, causing lock contention issues
[ https://issues.apache.org/jira/browse/SPARK-28102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen updated SPARK-28102:
-------------------------------
Summary: Failed LZ4 JNI initialization is repeatedly re-attempted, causing lock contention issues (was: Add configuration for selecting LZ4 implementation (safe, unsafe, JNI))
> Failed LZ4 JNI initialization is repeatedly re-attempted, causing lock contention issues
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-28102
> URL: https://issues.apache.org/jira/browse/SPARK-28102
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Major
>
> Spark's use of {{lz4-java}} ends up calling {{Lz4Factory.fastestInstance}}, which attempts to load JNI libraries and falls back on Java implementations in case the JNI library cannot be loaded or initialized.
> I run Spark in a configuration where the JNI libraries don't work, so I'd like to configure LZ4 to not even attempt to use JNI code: if the JNI library loads but cannot be initialized then the fallback code path involves catching an exception and this is slow because the exception is thrown under a static initializer lock (leading to significant lock contention because the filling of stacktraces is done while holding this lock).
> I propose to introduce a {{spark.io.compression.lz4.factory}} configuration for selecting the LZ4 implementation, allowing users to disable the use of the JNI library without having to recompile Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org