[jira] [Created] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

["Ramakrishna (Jira)" <ji...@apache.org>]

Ramakrishna created SPARK-44135:
-----------------------------------

             Summary: Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location
                 Key: SPARK-44135
                 URL: https://issues.apache.org/jira/browse/SPARK-44135
             Project: Spark
          Issue Type: Improvement
          Components: Shuffle
    Affects Versions: 3.2.0, 3.4.0
            Reporter: Ramakrishna


In our production environment, _finalizeShuffleMerge_ processing took longer time (p90 is around 20s) than other PRC requests. This is due to _finalizeShuffleMerge_ invoking IO operations like truncate and file open/close.  

More importantly, processing this _finalizeShuffleMerge_ can block other critical lightweight messages like authentications, which can cause authentication timeout as well as fetch failures. Those timeout and fetch failures affect the stability of the Spark job executions. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org