You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "rafafrdz (via GitHub)" <gi...@apache.org> on 2026/03/29 07:57:10 UTC
[PR] [SPARK-52428] Add DataFrame::to_local_iterator() for batch-by-batch results [spark-connect-rust]
rafafrdz opened a new pull request, #32:
URL: https://github.com/apache/spark-connect-rust/pull/32
## Summary
- Add `SparkConnectClient::to_arrow_batches()` that returns individual `Vec<RecordBatch>` without concatenating
- Add `DataFrame::to_local_iterator()` that returns batches individually, avoiding OOM on large datasets
- Unlike `collect()` which concatenates all batches into one `RecordBatch`, this preserves batch boundaries
## Test plan
- [x] `cargo build` passes
- [x] `cargo fmt -- --check` passes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52428] Add DataFrame::to_local_iterator() for batch-by-batch results [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #32: [SPARK-52428] Add DataFrame::to_local_iterator() for batch-by-batch results
URL: https://github.com/apache/spark-connect-rust/pull/32
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52428] Add DataFrame::to_local_iterator() for batch-by-batch results [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #32:
URL: https://github.com/apache/spark-connect-rust/pull/32#issuecomment-5403901051
Superseded by #50, which delivers the consolidated Rust core Spark Connect client (now merged to `master`) and incorporates this work. Closing to keep the queue focused on the drop-in Python wrapper (#51). Thanks for the contribution — please reopen if anything here isn't covered by #50.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org