You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/04/10 22:09:30 UTC
[PR] [SPARK-56253][PYTHON][CONNECT][FOLLOW-UP] Reject multi-column DataFrame input in spark.read.json [spark]
Yicong-Huang opened a new pull request, #55301:
URL: https://github.com/apache/spark/pull/55301
### What changes were proposed in this pull request?
Follow-up to #55097. Reject multi-column DataFrame input in `spark.read.json()` explicitly instead of silently using the first column and dropping the rest.
Also renames error conditions and methods from `PARSE_INPUT_*` to `DATAFRAME_INPUT_*` since these are query compilation errors, not parse errors.
Changes:
- Add `DATAFRAME_INPUT_NOT_SINGLE_COLUMN` error condition for multi-column (and zero-column) DataFrame input
- Rename `PARSE_INPUT_NOT_STRING_TYPE` to `DATAFRAME_INPUT_NOT_STRING_TYPE`
- Rename Scala methods from `parseInputNot*Error` to `dataframeInputNot*Error`
- Update tests to expect errors on multi-column and zero-column input
### Why are the changes needed?
Per review feedback on #55097 from cloud-fan and zhengruifeng: silently dropping columns is an anti-pattern. Multi-column DataFrame input should fail explicitly.
### Does this PR introduce _any_ user-facing change?
Yes. `spark.read.json(df)` now raises `DATAFRAME_INPUT_NOT_SINGLE_COLUMN` when the input DataFrame has more than one column (previously it silently used only the first column). Zero-column input now also raises `DATAFRAME_INPUT_NOT_SINGLE_COLUMN` instead of `PARSE_INPUT_NOT_STRING_TYPE`.
### How was this patch tested?
Updated existing tests in `test_datasources.py` (classic) and `test_connect_readwriter.py` (Connect) to verify that multi-column and zero-column input raises the expected error.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56253][PYTHON][CONNECT][FOLLOW-UP] Reject multi-column DataFrame input in spark.read.json [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #55301: [SPARK-56253][PYTHON][CONNECT][FOLLOW-UP] Reject multi-column DataFrame input in spark.read.json
URL: https://github.com/apache/spark/pull/55301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56253][PYTHON][CONNECT][FOLLOW-UP] Reject multi-column DataFrame input in spark.read.json [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #55301:
URL: https://github.com/apache/spark/pull/55301#issuecomment-4232853087
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56253][PYTHON][CONNECT][FOLLOW-UP] Reject multi-column DataFrame input in spark.read.json [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #55301:
URL: https://github.com/apache/spark/pull/55301#issuecomment-4227088851
cc @zhengruifeng @hvanhovell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org