You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2025/05/07 03:30:01 UTC
(spark) branch master updated: [SPARK-52022][PYTHON] Add additional check in default method for SparkThrowable.getQueryContext
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 39f6d25d8640 [SPARK-52022][PYTHON] Add additional check in default method for SparkThrowable.getQueryContext
39f6d25d8640 is described below
commit 39f6d25d8640be5f0f55fa0e5cfef89af0d13b3d
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Wed May 7 12:29:48 2025 +0900
[SPARK-52022][PYTHON] Add additional check in default method for SparkThrowable.getQueryContext
### What changes were proposed in this pull request?
This PR proposes to add an additional check the default method for `SparkThrowable.getQueryContext`.
### Why are the changes needed?
Yes, it breaks Spark 4.0. It is described below.
### Does this PR introduce _any_ user-facing change?
Yes. Seems like Py4J does not support default method access. So if `getQueryContext` is not explicitly overridden, it throws an exception like:
```
py4j.protocol.Py4JError: An error occurred while calling o91.getQueryContext. Trace:
py4j.Py4JException: Method getQueryContext([]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:321)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:329)
at py4j.Gateway.invoke(Gateway.java:274)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:184)
at py4j.ClientServerConnection.run(ClientServerConnection.java:108)
at java.base/java.lang.Thread.run(Thread.java:840)
```
### How was this patch tested?
Manually tested.
### Was this patch authored or co-authored using generative AI tooling?
No,
Closes #50807 from HyukjinKwon/SPARK-52022.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/errors/exceptions/captured.py | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/python/pyspark/errors/exceptions/captured.py b/python/pyspark/errors/exceptions/captured.py
index ba5c2601a6a6..85e01a1778af 100644
--- a/python/pyspark/errors/exceptions/captured.py
+++ b/python/pyspark/errors/exceptions/captured.py
@@ -122,8 +122,10 @@ class CapturedException(PySparkException):
assert SparkContext._gateway is not None
gw = SparkContext._gateway
- if self._origin is not None and is_instance_of(
- gw, self._origin, "org.apache.spark.SparkThrowable"
+ if (
+ self._origin is not None
+ and is_instance_of(gw, self._origin, "org.apache.spark.SparkThrowable")
+ and hasattr(self._origin, "getMessageParameters")
):
return dict(self._origin.getMessageParameters())
else:
@@ -135,8 +137,10 @@ class CapturedException(PySparkException):
assert SparkContext._gateway is not None
gw = SparkContext._gateway
- if self._origin is not None and is_instance_of(
- gw, self._origin, "org.apache.spark.SparkThrowable"
+ if (
+ self._origin is not None
+ and is_instance_of(gw, self._origin, "org.apache.spark.SparkThrowable")
+ and hasattr(self._origin, "getSqlState")
):
return self._origin.getSqlState()
else:
@@ -149,8 +153,10 @@ class CapturedException(PySparkException):
assert SparkContext._gateway is not None
gw = SparkContext._gateway
- if self._origin is not None and is_instance_of(
- gw, self._origin, "org.apache.spark.SparkThrowable"
+ if (
+ self._origin is not None
+ and is_instance_of(gw, self._origin, "org.apache.spark.SparkThrowable")
+ and hasattr(self._origin, "getMessageParameters")
):
errorClass = self._origin.getCondition()
messageParameters = self._origin.getMessageParameters()
@@ -170,8 +176,10 @@ class CapturedException(PySparkException):
assert SparkContext._gateway is not None
gw = SparkContext._gateway
- if self._origin is not None and is_instance_of(
- gw, self._origin, "org.apache.spark.SparkThrowable"
+ if (
+ self._origin is not None
+ and is_instance_of(gw, self._origin, "org.apache.spark.SparkThrowable")
+ and hasattr(self._origin, "getQueryContext")
):
contexts: List[BaseQueryContext] = []
for q in self._origin.getQueryContext():
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org