You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "caldempsey (via GitHub)" <gi...@apache.org> on 2025/07/12 16:44:24 UTC
[PR] [SPARK-52780] Implement ToLocalRowIterator (ToLocalIterator equivalent) [spark-connect-go]
caldempsey opened a new pull request, #151:
URL: https://github.com/apache/spark-connect-go/pull/151
### What changes were proposed in this pull request?
* **Incremental result streaming for the Spark Connect Go client**
* Introduces a new `types.RowIterator` interface plus a concrete `arrowRowIterator` that streams rows out of an Arrow table one-by-one.
* Adds `ReadArrowTableToIterator(arrow.Table)` in `types/arrow.go`, the lazy counterpart to `ReadArrowTableToRows`.
* Extends the `DataFrame` API with **`ToLocalRowIterator(ctx)`** (renamed from the initially-suggested `ToLocalIterator` to make it crystal-clear that *rows* are iterated from the position of selection in your IDE or favorite text editor, I still think the naming convention doesn't make clear this is incrementally, as they arrive though).
* **Tests**
* New unit-tests in `types/arrow_test.go` cover:
* partial consumption (fetch a single row, then the remainder), and
* equivalence between the lazy iterator and eager conversion.
There are **no modifications to existing public methods or proto contracts** – only additive, backwards-compatible extensions. I would suggest we merge this behavior with the current Collect function though.
### Why are the changes needed?
The Go client could previously retrieve results only through `Collect()`, which loads every row into memory before returning. This is impractical for large OLAP workloads that routinely return GBs of data and prevents lightweight services (API gateways, lambda-style functions, etc.) from using Spark Connect efficiently.
Scala/Java clients solve this with `toLocalIterator()`. This PR brings the same capability to Go:
* **Constant-memory iteration** – rows are deserialised batch-by-batch and released immediately after consumption.
* **Back-pressure friendly** – external consumers can pull at their own pace (e.g. stream over gRPC/HTTP).
* **API parity** with the other Spark Connect language bindings.
* **Clearer naming** – the new method is called `ToLocalRowIterator` so users instantly understand they are iterating **row by row**.
### Does this PR introduce *any* user-facing change?
Yes, a **new opt-in API**:
```go
it, err := df.ToLocalRowIterator(ctx)
for {
row, err := it.Next()
if err == io.EOF { break }
// process row ...
}
```
Again.. Existing code that uses `Collect()` or `ToArrow()` continues to work unchanged.
### How was this patch tested?
* **Unit tests** in `types/arrow_test.go` verify:
* Streaming access returns rows in the correct order and signals `io.EOF` properly.
* The set of rows produced by the iterator is exactly equal to the rows produced by the existing eager converter.
* The full Go test suite (`go test ./...`) passes locally on Linux with Go 1.21.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52780] Implement ToLocalRowIterator (ToLocalIterator equivalent) [spark-connect-go]
Posted by "caldempsey (via GitHub)" <gi...@apache.org>.
caldempsey commented on PR #151:
URL: https://github.com/apache/spark-connect-go/pull/151#issuecomment-3065952826
Hm. This is failing my E2E test. I suspect the Arrow table is being fully materialised instead of also writing the data batches out. This is something I've handled before in a Databricks implementation. Looking into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52780] Implement ToLocalRowIterator (ToLocalIterator equivalent) [spark-connect-go]
Posted by "caldempsey (via GitHub)" <gi...@apache.org>.
caldempsey closed pull request #151: [SPARK-52780] Implement ToLocalRowIterator (ToLocalIterator equivalent)
URL: https://github.com/apache/spark-connect-go/pull/151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52780] Implement ToLocalRowIterator (ToLocalIterator equivalent) [spark-connect-go]
Posted by "caldempsey (via GitHub)" <gi...@apache.org>.
caldempsey commented on PR #151:
URL: https://github.com/apache/spark-connect-go/pull/151#issuecomment-3066284840
I've updated the code, think its easier for me to just force push here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org