You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "marcuslin123 (via GitHub)" <gi...@apache.org> on 2026/07/08 01:45:29 UTC
[PR] [SPARK-57965][CONNECT][PYSPARK] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE parameter name mismatch [spark]
marcuslin123 opened a new pull request, #57108:
URL: https://github.com/apache/spark/pull/57108
### What changes were proposed in this pull request?
Fix the `messageParameters` key in `SparkConnectGraphElementRegistry.register_output` from `output_type` to `dataset_type` at both raise sites, matching the `UNSUPPORTED_PIPELINES_DATASET_TYPE` error template which interpolates `<dataset_type>`.
### Why are the changes needed?
The mismatch between the template placeholder (`<dataset_type>`) and the provided parameter key (`output_type`) causes `ErrorClassesReader.get_error_message` to fire an assertion, replacing the intended `PySparkTypeError` with an opaque `AssertionError`. Users see a confusing internal crash instead of the helpful "Unsupported pipelines dataset type: X" message.
### Does this PR introduce _any_ user-facing change?
Yes. Users who pass an unsupported output type to Declarative Pipelines will now see:
```
PySparkTypeError: [UNSUPPORTED_PIPELINES_DATASET_TYPE] Unsupported pipelines dataset type: SomeClass.
```
Instead of the previous opaque `AssertionError`.
### How was this patch tested?
Added a regression test in `test_graph_element_registry.py` that constructs `PySparkTypeError` with `UNSUPPORTED_PIPELINES_DATASET_TYPE` and verifies the error message is produced without assertion failures.
### Was this patch authored or co-authored using generative AI tooling?
Generative AI tooling (Claude Code) was used as an assistive tool for implementation guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57965][CONNECT][PYSPARK] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE parameter name mismatch [spark]
Posted by "marcuslin123 (via GitHub)" <gi...@apache.org>.
marcuslin123 commented on PR #57108:
URL: https://github.com/apache/spark/pull/57108#issuecomment-5004063436
Thanks @uros-b for pointing this out — I hadn't seen #57005. Closing this in favor of the earlier PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57965][CONNECT][PYSPARK] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE parameter name mismatch [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57108:
URL: https://github.com/apache/spark/pull/57108#issuecomment-4981204856
I would propose to consolidate on the earlier PR, and close the current one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57965][CONNECT][PYSPARK] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE parameter name mismatch [spark]
Posted by "marcuslin123 (via GitHub)" <gi...@apache.org>.
marcuslin123 closed pull request #57108: [SPARK-57965][CONNECT][PYSPARK] Fix UNSUPPORTED_PIPELINES_DATASET_TYPE parameter name mismatch
URL: https://github.com/apache/spark/pull/57108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org