You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2019/06/19 22:27:00 UTC
[jira] [Resolved] (SPARK-28102) Failed LZ4 JNI initialization is
repeatedly re-attempted, causing lock contention issues
[ https://issues.apache.org/jira/browse/SPARK-28102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen resolved SPARK-28102.
--------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 24905
[https://github.com/apache/spark/pull/24905]
> Failed LZ4 JNI initialization is repeatedly re-attempted, causing lock contention issues
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-28102
> URL: https://issues.apache.org/jira/browse/SPARK-28102
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Major
> Fix For: 3.0.0
>
>
> Spark's use of {{lz4-java}} ends up calling {{LZ4Factory.fastestInstance}}, which attempts to load JNI libraries and falls back on Java implementations in case the JNI library cannot be loaded or initialized.
> Unfortunately, the LZ4 library does not cache the fact that the JNI library failed to load, so every call to {{LZ4Factory.fastestInstance}} ends up repeating this failed load. These initialization attempts are performed in a static synchronized method, so exceptions from failures are thrown while holding a global lock and this causes lock-contention performance issues.
> To avoid this problem, we can cache the LZ4Factory ourselves (since it's thread-safe), thereby caching the outcome of the failed load and avoiding performance problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org