You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2019/10/24 01:18:00 UTC
[jira] [Assigned] (SPARK-29576) Use Spark's CompressionCodec for
Ser/Deser of MapOutputStatus
[ https://issues.apache.org/jira/browse/SPARK-29576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun reassigned SPARK-29576:
-------------------------------------
Assignee: DB Tsai
> Use Spark's CompressionCodec for Ser/Deser of MapOutputStatus
> -------------------------------------------------------------
>
> Key: SPARK-29576
> URL: https://issues.apache.org/jira/browse/SPARK-29576
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 2.4.4
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Fix For: 3.0.0
>
>
> Instead of using ZStd codec directly, we use Spark's CompressionCodec which wraps ZStd codec in buffered stream to avoid overhead excessive of JNI call while trying to compress small amount of data.
> Also, by using Spark's CompressionCodec, we can easily to make it configurable in the future if needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org