[GitHub] [iceberg] fjetter commented on issue #3283: [Python] gist examples for iceberg python library

[GitBox <gi...@apache.org>]

fjetter commented on issue #3283:
URL: https://github.com/apache/iceberg/issues/3283#issuecomment-941232976


   I created a short jupyter notebook (download the notebook + `conda install dask jupyterlab pyarrow` and you're good to go) about how the dask API currently reads/writes a parquet dataset. The notebook generates random data and roundtrips to a parquet dataset.
   The API is basically two module level functions `to_parquet` and `from_parquet` with a lot of arguments. Some of the arguments are for the query planning while others are for the compute engine, storage, etc. Behind the scenes we're disentangling things.
   
   An easy iceberg integration could simply be to provide `iceberg` as an engine instead of `pyarrow` and add some more keywords like `snapshot_id`, etc. _Formally_ that's incorrect but it would be an easy way for us to patch things in :)
   
   The `dask.DataFrame` object itself exposes high level aggregation and shaping methods but, for instance, you can also access individual files/splits/tasks (whatever your terminology is). There is also a `DaskDataframe.describe` method but that would currently actually load all data and calculate statistics. We do want to improve our API (also for simple stuff like `len`/ count) such that this accesses metadata instead of payload data but that's actually quite a bit of work and will still take a while.
   
   https://docs.dask.org/en/latest/generated/dask.dataframe.read_parquet.html#dask.dataframe.read_parquet
   https://docs.dask.org/en/latest/generated/dask.dataframe.to_parquet.html#dask.dataframe.to_parquet
   
   https://gist.github.com/fjetter/86d009f9365dbd4864aa13946f69b758


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org