[I] Expose simplified Arrow storage for Python stream export [sedona-db]

["camden-lowrance (via GitHub)" <gi...@apache.org>]

camden-lowrance opened a new issue, #869:
URL: https://github.com/apache/sedona-db/issues/869

   Some Python callers consume SedonaDB results through the Arrow stream interface instead of collecting a full table with `to_pandas()` / `to_arrow_table()`. This is useful for large results where callers need to process record batches incrementally.
   
   Today, the public stream path preserves Arrow storage types. That is efficient, but it means callers may receive newer storage such as `BinaryView` or `Utf8View`. Callers that need simpler Arrow storage can work around this in SQL with `sd_simplifystorage(...)`, but this requires them to know which columns need simplification.
   
   SedonaDB already has an internal path that can simplify all columns while exporting a stream:
   
   ```python
   self._impl.to_stream(self._ctx, simplify=True)
   ```
   
   It would be useful to expose this capability through the Python API so streaming callers can request simplified Arrow storage at the export boundary, without adding column-specific SQL casts or `sd_simplifystorage(...)` calls.
   
   Possible API shape, naming TBD:
   
   ```python
   reader = df.to_record_batch_reader(simplify=True)
   ```
   
   or another public streaming method that maps to the existing simplified stream behavior.
   
   Related context: #864 and #868.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Expose simplified Arrow storage for Python stream export [sedona-db]

["paleolimbot (via GitHub)" <gi...@apache.org>]

paleolimbot closed issue #869: Expose simplified Arrow storage for Python stream export
URL: https://github.com/apache/sedona-db/issues/869


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org