You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2025/08/07 04:11:00 UTC
[jira] [Commented] (SPARK-53048) Include PySpark exception reasons in SparkUserAppException
[ https://issues.apache.org/jira/browse/SPARK-53048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012468#comment-18012468 ]
Dongjoon Hyun commented on SPARK-53048:
---------------------------------------
Thank you for filing a JIRA issue, [~kazuyukitanimura].
It sounds like two theme here.
1. We may want to extend `SparkUserAppException` to have `cause: Throwable` parameter. It's doable.
2. However, how did you fill the `Throwable` in the above code? `process.waitFor` simply return `exitCode` only.
Could you elaborate on where did you get the `Throwable` instance from the non-Java process like Python to propagate to Java?
> Include PySpark exception reasons in SparkUserAppException
> ----------------------------------------------------------
>
> Key: SPARK-53048
> URL: https://issues.apache.org/jira/browse/SPARK-53048
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.1.0
> Reporter: Kazuyuki Tanimura
> Priority: Major
>
> Currently the exit code of Python process failure is passed to SparkUserAppException.
> In particular this piece of code [https://github.com/apache/spark/blob/3bf3696461ab18e6b2507f3f1a2729568e4a6832/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala#L127]
> ```
> try {
> val process = builder.start()
>
> new RedirectThread(process.getInputStream, System.out, "redirect output").start()
>
> val exitCode = process.waitFor()
> if (exitCode != 0)
> { throw new SparkUserAppException(exitCode) }
> } finally
> { gatewayServer.foreach(_.shutdown()) }
> ```
> Although logger can be set for PySpark, it would be more user friendly if we can include more explicit reasons or Python stacktrace to SparkUserAppException, so that users do not have to look into the other part of the logs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org