[GitHub] [iceberg] TomAugspurger commented on issue #5800: Integrate pyiceberg with Dask

["TomAugspurger (via GitHub)" <gi...@apache.org>]

TomAugspurger commented on issue #5800:
URL: https://github.com/apache/iceberg/issues/5800#issuecomment-1553478936

   > Generally, should this be part of the Dask library instead of PyIceberg?
   
   I was wondering that too. I think either would be sensible, but I'd lean slightly towards putting the implementation in Dask (and pyiceberg could add a `.to_dask_dataframe` to mirror `to_pandas` if desired).
   
   That said, the current implementation does use a private `_file_to_table`. I think pyiceberg would need to add a public method that achieves that before Dask could use it. And given the similarity to `pyiceberg.io.pyarrow`, maybe we could start with some refactoring to add some methods used by both the `pyarrow` and `dask.dataframe` implementations?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org