[jira] [Created] (SPARK-39752) Spark job failed with 10M rows data with Broken pipe error

["SHOBHIT SHUKLA (Jira)" <ji...@apache.org>]

SHOBHIT SHUKLA created SPARK-39752:
--------------------------------------

             Summary: Spark job failed with 10M rows data with Broken pipe error
                 Key: SPARK-39752
                 URL: https://issues.apache.org/jira/browse/SPARK-39752
             Project: Spark
          Issue Type: Bug
          Components: PySpark, Spark Core
    Affects Versions: 3.2.1, 3.0.3
            Reporter: SHOBHIT SHUKLA
             Fix For: 3.0.2


Spark job failed with 10M rows data with Broken pipe error. Same spark job was working previously with the settings "executor_cores": 1, "executor_memory": 1, "driver_cores": 1, "driver_memory": 1. where as the same job is failing with spark settings in 3.0.3 and 3.2.1.

Major symptoms (slowness, timeout, out of memory as examples): Spark job is failing with the error java.net.SocketException: Broken pipe (Write failed)


Here are the spark settings information which is working on Spark 3.0.3 and 3.2.1 : "executor_cores": 4, "executor_memory": 4, "driver_cores": 4, "driver_memory": 4.. The spark job doesn't consistently works with the above settings. Some times, need to increase the cores and memory.





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org