You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/03/27 10:12:23 UTC
[PR] [SPARK-56257][PYTHON][CONNECT] Support DataFrame input for spark.read.json/csv/xml [spark]
Yicong-Huang opened a new pull request, #55057:
URL: https://github.com/apache/spark/pull/55057
### What changes were proposed in this pull request?
Allow `spark.read.json()`, `spark.read.csv()`, and `spark.read.xml()` to accept a DataFrame with a single string column as input. Connect supports JSON and CSV only — XML will be added in a follow-up PR after extending the `Parse` proto.
### Why are the changes needed?
Parsing in-memory text data into a structured DataFrame currently requires `sc.parallelize()`, which is unavailable on Spark Connect. This is the inverse of `DataFrame.toJSON()`.
### Does this PR introduce _any_ user-facing change?
Yes. `spark.read.json()`, `csv()`, and `xml()` now accept a single-string-column DataFrame as input.
### How was this patch tested?
7 new tests: 4 classic (JSON, JSON+schema, CSV, XML) and 3 Connect (JSON, CSV, XML-unsupported).
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56257][PYTHON][CONNECT] Support DataFrame input for spark.read.json/csv/xml [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #55057:
URL: https://github.com/apache/spark/pull/55057#issuecomment-4156555266
Closing this POC PR. Split into individual PRs per JIRA sub-task under SPARK-55227:\n- SPARK-56253 (JSON): #55097\n- SPARK-56255 (CSV): upcoming\n- SPARK-56254 (XML): upcoming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56257][PYTHON][CONNECT] Support DataFrame input for spark.read.json/csv/xml [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #55057:
URL: https://github.com/apache/spark/pull/55057#issuecomment-4141715669
cc @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56257][PYTHON][CONNECT] Support DataFrame input for spark.read.json/csv/xml [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #55057: [SPARK-56257][PYTHON][CONNECT] Support DataFrame input for spark.read.json/csv/xml
URL: https://github.com/apache/spark/pull/55057
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org