You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/02/03 07:19:13 UTC
[PR] [SPARK-55336][PYTHON] Let createDF use create_batch logic for decoupling [spark]
Yicong-Huang opened a new pull request, #54111:
URL: https://github.com/apache/spark/pull/54111
### What changes were proposed in this pull request?
This PR duplicates the pandas-to-Arrow batch conversion logic in `ArrowStreamPandasSerializer` to decouple it.
- `create_arrow_array_from_pandas()` - converts a pandas Series to Arrow Array
- `create_arrow_batch_from_pandas()` - converts a list of (series, spark_type) tuples to Arrow RecordBatch
Both `_create_from_pandas_with_arrow` (classic Spark) and `createDataFrame` (Spark Connect) now use these standalone functions directly with `ArrowStreamSerializer`, instead of depending on `ArrowStreamPandasSerializer`.
### Why are the changes needed?
For better decoupling. Previously, `createDataFrame` had to instantiate `ArrowStreamPandasSerializer` just to call its `_create_batch` method.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55336][PYTHON] Let createDF use create_batch logic for decoupling [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54111:
URL: https://github.com/apache/spark/pull/54111#issuecomment-3839548371
## JIRA Issue Information
=== Sub-task SPARK-55336 ===
Summary: Factor out ArrowStreamPandasSerializer._create_batch logic for createDataFrame
Assignee: None
Status: Open
Affected: ["4.2.0"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55336][PYTHON] Let createDF use create_batch logic for decoupling [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #54111: [SPARK-55336][PYTHON] Let createDF use create_batch logic for decoupling
URL: https://github.com/apache/spark/pull/54111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55336][PYTHON] Let createDF use create_batch logic for decoupling [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #54111:
URL: https://github.com/apache/spark/pull/54111#issuecomment-3844084568
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org