You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Emil Ejbyfeldt (Jira)" <ji...@apache.org> on 2022/10/25 12:51:00 UTC

[jira] [Created] (SPARK-40912) Large overhead of Exceptions in DeserializationStream

Emil Ejbyfeldt created SPARK-40912:
--------------------------------------

             Summary: Large overhead of Exceptions in DeserializationStream 
                 Key: SPARK-40912
                 URL: https://issues.apache.org/jira/browse/SPARK-40912
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 3.3.0
            Reporter: Emil Ejbyfeldt


The interface of DeserializationStream forces implementation to raise EOFException to indicate that there is no more data. And for the KryoDeserializtionStream it even worse since the kryo library does not raise EOFException we pay for the price of two exceptions for each stream. For large shuffles with lots of small stream this is quite a bit large overhead (seen couple % of cpu time).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org