You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "angerszhu (Jira)" <ji...@apache.org> on 2021/10/26 10:26:00 UTC
[jira] [Commented] (SPARK-35181) Use zstd for
spark.io.compression.codec by default
[ https://issues.apache.org/jira/browse/SPARK-35181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434241#comment-17434241 ]
angerszhu commented on SPARK-35181:
-----------------------------------
[~dongjoon] This issue resolve when we upgrade the zstd version, but another issue happened.
{code:java}
2021-10-25 13:42:01 WARN SparkConf:69 - The configuration key 'spark.blacklist.application.fetchFailure.enabled' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.application.fetchFailure.enabled
2021-10-25 13:42:01 WARN SparkConf:69 - The configuration key 'spark.blacklist.enabled' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.enabled
2021-10-25 13:42:01 WARN SparkConf:69 - The configuration key 'spark.blacklist.killBlacklistedExecutors' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.killExcludedExecutors
2021-10-25 13:42:02 INFO EventMetricSparkPlugin:20 - Start to register event process metric plugin.
2021-10-25 13:42:10 INFO deprecation:1398 - No unit for dfs.client.datanode-restart.timeout(30) assuming SECONDS
2021-10-25 13:42:10 INFO deprecation:1398 - No unit for dfs.client.datanode-restart.timeout(30) assuming SECONDS
2021-10-25 13:42:23 INFO CodecPool:184 - Got brand-new decompressor [.zst]
2021-10-25 13:42:23 INFO CodecPool:184 - Got brand-new decompressor [.zst]
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f4017bb3112, pid=58809, tid=0x00007f402cffe700
#
# JRE version: OpenJDK Runtime Environment (8.0_232-b09) (build 1.8.0_232-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.232-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libzstd-jni-1.5.0-28889732549921047792.so+0xc6112]
#
# Core dump written. Default location: /mnt/ssd/0/yarn/nm-local-dir/usercache/staging_data_trafficmart/appcache/application_1632999515383_3679724/container_e238_1632999515383_3679724_02_000002/core or core.58809
#
# An error report file with more information is saved as:
# /mnt/ssd/0/yarn/nm-local-dir/usercache/staging_data_trafficmart/appcache/application_1632999515383_3679724/container_e238_1632999515383_3679724_02_000002/hs_err_pid58809.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
{code}
Seems zstd so unstable? Or it's related to our zstd env problem?
> Use zstd for spark.io.compression.codec by default
> --------------------------------------------------
>
> Key: SPARK-35181
> URL: https://issues.apache.org/jira/browse/SPARK-35181
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: Dongjoon Hyun
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org