You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2026/10/13 11:06:00 UTC
[jira] [Assigned] (SPARK-59909) PySpark raises internal errors instead of intended messages due to undefined error conditions and mismatched message parameters
[ https://issues.apache.org/jira/browse/SPARK-59909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-59909:
-----------------------------------
Assignee: Jubin Soni
> PySpark raises internal errors instead of intended messages due to undefined error conditions and mismatched message parameters
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59909
> URL: https://issues.apache.org/jira/browse/SPARK-59909
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.1.0, 4.0.0, 4.2.0
> Reporter: Jubin Soni
> Assignee: Jubin Soni
> Priority: Minor
> Labels: pull-request-available
>
> Several PySpark call sites raise a PySparkException with an error condition that does not exist in python/pyspark/errors/error-conditions.json, or with messageParameters that do not match the condition's template. Constructing the exception itself fails, so the user gets an unrelated internal error instead of the intended message, often with a different exception type (e.g. ValueError instead of TypeError).
> h3. Undefined error conditions (fails with "ValueError: Cannot find main error class ...")
> * NOT_STR - DataStreamReader.changes() (pyspark/sql/streaming/readwriter.py). NOT_STR was consolidated into NOT_EXPECTED_TYPE by SPARK-56179 while this new usage was added concurrently.
> * VARIANT_SIZE_LIMIT_EXCEEDED - pyspark/sql/variant_utils.py (3 places)
> * INVALID_OPERATION_UUID_ID - pyspark/sql/connect/client/core.py
> * UNSUPPORTED_FILTER - pyspark/sql/worker/data_source_pushdown_filters.py
> h3. Mismatched message parameters (fails with "AssertionError: Undefined error message parameter ...")
> * DATA_SOURCE_EXTRANEOUS_FILTERS - pyspark/sql/worker/data_source_pushdown_filters.py passes extra "input" and "extraneous" parameters
> * NO_ACTIVE_SESSION - pyspark/sql/connect/dataframe.py passes an extra "operator" parameter
> h3. Reproduction
> {code:python}
> >>> spark.readStream.changes(123)
> ValueError: Cannot find main error class 'NOT_STR'
> {code}
> Expected: PySparkTypeError saying that `tableName` should be a str.
> The existing test for DATA_SOURCE_EXTRANEOUS_FILTERS (test_python_datasource.py) does not catch this, because it only asserts that the condition name appears in the exception text, which is also true of the AssertionError.
> The JVM side guards against this in SparkThrowableSuite, but PySpark has no equivalent check.
> h3. Proposed fix
> Fix each call site, and add a test that scans PySpark sources for errorClass literals and verifies that each one is defined and that its literal messageParameters keys match the template.
> UNSUPPORTED_PIPELINES_DATASET_TYPE has the same kind of problem but is already tracked in SPARK-57965, so it is out of scope here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org