Re: [I] Support writing Arrow RecordBatchReader or Scanner to Iceberg tables [iceberg-python]

["jayceslesar (via GitHub)" <gi...@apache.org>]

jayceslesar commented on issue #2152:
URL: https://github.com/apache/iceberg-python/issues/2152#issuecomment-3067178339

   I think this is really hard to do without fully materializing the table and would take a very long time to complete if you are writing a large stream of record batches.
   
   We would have to process the entire stream of batches here https://github.com/apache/iceberg-python/blob/dc439402e33d642c9a0c3261f10c078017d6566e/pyiceberg/io/pyarrow.py#L2711 at least, and then re-sink everything to files per partition, then process all of those temporary files that now need to be cleaned up as a stream.
   
   You are correct that it does not work with overwrite in its current state. I would really suggest just using a bigger compute runtime if you have to use pyiceberg for an operation like this otherwise use spark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org