[jira] [Updated] (SPARK-39752) Spark job failed with 10M rows data with Broken pipe error

["SHOBHIT SHUKLA (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-39752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

SHOBHIT SHUKLA updated SPARK-39752:
-----------------------------------
    Attachment:     (was: Failed_spark_job_3.0.3.txt)

> Spark job failed with 10M rows data with Broken pipe error
> ----------------------------------------------------------
>
>                 Key: SPARK-39752
>                 URL: https://issues.apache.org/jira/browse/SPARK-39752
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, Spark Core
>    Affects Versions: 3.0.3, 3.2.1
>            Reporter: SHOBHIT SHUKLA
>            Priority: Major
>             Fix For: 3.0.2
>
>
> Spark job failed with 10M rows data with Broken pipe error. Same spark job was working previously with the settings "executor_cores": 1, "executor_memory": 1, "driver_cores": 1, "driver_memory": 1. where as the same job is failing with spark settings in 3.0.3 and 3.2.1.
> Major symptoms (slowness, timeout, out of memory as examples): Spark job is failing with the error java.net.SocketException: Broken pipe (Write failed)
> Here are the spark settings information which is working on Spark 3.0.3 and 3.2.1 : "executor_cores": 4, "executor_memory": 4, "driver_cores": 4, "driver_memory": 4.. The spark job doesn't consistently works with the above settings. Some times, need to increase the cores and memory.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org