You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anxkhn (via GitHub)" <gi...@apache.org> on 2026/07/05 08:45:31 UTC
[PR] [PYTHON][CONNECT] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE error parameter name [spark]
anxkhn opened a new pull request, #57005:
URL: https://github.com/apache/spark/pull/57005
### What changes were proposed in this pull request?
`SparkConnectGraphElementRegistry.register_output`
(`python/pyspark/pipelines/spark_connect_graph_element_registry.py`) has two
`else`-branch raise sites that report an unsupported pipeline output type. Both
raised:
```python
raise PySparkTypeError(
errorClass="UNSUPPORTED_PIPELINES_DATASET_TYPE",
messageParameters={"output_type": type(output).__name__},
)
```
but the `UNSUPPORTED_PIPELINES_DATASET_TYPE` template in
`python/pyspark/errors/error-conditions.json` interpolates `<dataset_type>`:
```
"Unsupported pipelines dataset type: <dataset_type>."
```
`ErrorClassesReader.get_error_message` (`python/pyspark/errors/utils.py`) asserts
that the set of template placeholders equals the set of provided message
parameters. Since `{"dataset_type"} != {"output_type"}`, the assertion fired
inside `PySparkException.__init__`, so the intended `PySparkTypeError` was
replaced by an opaque `AssertionError`.
This PR renames the message parameter key from `"output_type"` to
`"dataset_type"` at both raise sites (the value, `type(output).__name__`, is
unchanged), so it matches the template placeholder. The error class is
referenced only at these two Python sites plus the template, so the change is
fully local. A regression test is added.
### Why are the changes needed?
Registering a pipeline output of an unsupported type is meant to surface a clear
`PySparkTypeError` with condition `UNSUPPORTED_PIPELINES_DATASET_TYPE`. Because of
the parameter-name mismatch it instead raised an unrelated `AssertionError` from
the error-formatting layer, hiding the real cause from the user:
```
AssertionError: Undefined error message parameter for error class:
UNSUPPORTED_PIPELINES_DATASET_TYPE. Parameters: {'output_type': 'Output'}
```
The intended error is never constructed, so callers cannot catch
`PySparkTypeError` or read the intended message.
### Does this PR introduce _any_ user-facing change?
Yes, an error-behavior change on the failure path only. Registering a pipeline
output whose type is unsupported now raises the intended, documented error.
Before:
```
AssertionError: Undefined error message parameter for error class:
UNSUPPORTED_PIPELINES_DATASET_TYPE. Parameters: {'output_type': 'Output'}
```
After:
```
pyspark.errors.exceptions.base.PySparkTypeError:
[UNSUPPORTED_PIPELINES_DATASET_TYPE] Unsupported pipelines dataset type: Output.
```
This is a fix relative to the unreleased `master` behavior (the affected
`messageParameters={"output_type": ...}` code path); no released Spark version
behavior changes here.
### How was this patch tested?
Added `SparkConnectPipelinesTest.test_register_output_unsupported_dataset_type`
in `python/pyspark/pipelines/tests/test_spark_connect.py`, which builds a
`SparkConnectGraphElementRegistry`, calls `register_output` with a bare `Output`
(an unsupported type, since it is not `Table`/`TemporaryView`/`Sink`), and
asserts `PySparkTypeError` with `getCondition() ==
"UNSUPPORTED_PIPELINES_DATASET_TYPE"` and `getMessageParameters() ==
{"dataset_type": "Output"}`.
The test extends `ReusedConnectTestCase`, so CI runs it against a real Spark
Connect server. Verified locally against a live Connect server:
- New test on this change: `Ran 1 test ... OK`.
- Temporarily reverting the fix (restoring the `"output_type"` key) makes the
same test fail with the `AssertionError` above, confirming it is a genuine
RED -> GREEN regression test.
- Full `pyspark.pipelines.tests.test_spark_connect` module: `Ran 5 tests ... OK`,
no regressions.
Lint (repo-pinned versions): `black==26.3.1` `--check`, `ruff==0.14.8` `check`,
and `mypy==1.19.1` all pass on the two changed files.
### Was this patch authored or co-authored using generative AI tooling?
<!-- Anas: decide and fill this in before opening the PR. The ASF asks that if
generative AI tooling was used, you include "Generated-by: <tool> <version>".
See https://www.apache.org/legal/generative-tooling.html . -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57965][PYTHON][CONNECT] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE error parameter name [spark]
Posted by "anxkhn (via GitHub)" <gi...@apache.org>.
anxkhn commented on code in PR #57005:
URL: https://github.com/apache/spark/pull/57005#discussion_r3593027380
##########
python/pyspark/pipelines/tests/test_spark_connect.py:
##########
@@ -90,6 +93,20 @@ def st():
"INVALID_FLOW_QUERY_TYPE.BATCH_RELATION_FOR_STREAMING_TABLE", str(context.exception)
)
+ def test_register_output_unsupported_dataset_type(self):
+ registry = SparkConnectGraphElementRegistry(self.spark, "test_dataflow_graph_id")
+
+ unsupported_output = Output(
+ name="unsupported",
+ comment=None,
+ source_code_location=SourceCodeLocation(filename="test.py", line_number=1),
+ )
+ with self.assertRaises(PySparkTypeError) as context:
+ registry.register_output(unsupported_output)
+
+ self.assertEqual(context.exception.getCondition(), "UNSUPPORTED_PIPELINES_DATASET_TYPE")
+ self.assertEqual(context.exception.getMessageParameters(), {"dataset_type": "Output"})
+
Review Comment:
Added a CustomTable case, so the regression test now covers both raise sites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57965][PYTHON][CONNECT] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE error parameter name [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #57005:
URL: https://github.com/apache/spark/pull/57005#issuecomment-6391547264
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57965][PYTHON][CONNECT] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE error parameter name [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #57005: [SPARK-57965][PYTHON][CONNECT] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE error parameter name
URL: https://github.com/apache/spark/pull/57005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [PYTHON][CONNECT] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE error parameter name [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #57005:
URL: https://github.com/apache/spark/pull/57005#discussion_r3524957814
##########
python/pyspark/pipelines/tests/test_spark_connect.py:
##########
@@ -90,6 +93,20 @@ def st():
"INVALID_FLOW_QUERY_TYPE.BATCH_RELATION_FOR_STREAMING_TABLE", str(context.exception)
)
+ def test_register_output_unsupported_dataset_type(self):
+ registry = SparkConnectGraphElementRegistry(self.spark, "test_dataflow_graph_id")
+
+ unsupported_output = Output(
+ name="unsupported",
+ comment=None,
+ source_code_location=SourceCodeLocation(filename="test.py", line_number=1),
+ )
+ with self.assertRaises(PySparkTypeError) as context:
+ registry.register_output(unsupported_output)
+
+ self.assertEqual(context.exception.getCondition(), "UNSUPPORTED_PIPELINES_DATASET_TYPE")
+ self.assertEqual(context.exception.getMessageParameters(), {"dataset_type": "Output"})
+
Review Comment:
The regression test exercises only the second (outer else) raise site via a bare Output(). The first raise site (inside if isinstance(output, Table):, a Table subclass that is neither MaterializedView nor StreamingTable) carries the identical bug+fix but is untested; a class CustomTable(Table) assertion would close the gap and guard against a future regression on that branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org