You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/12 05:53:02 UTC
[GitHub] [arrow] jhostetler opened a new issue, #33624: [GCS] Scanning hive-partitioned dataset results in orders of magnitude more network traffic than it should
jhostetler opened a new issue, #33624:
URL: https://github.com/apache/arrow/issues/33624
### Describe the bug, including details regarding any error messages, version, and platform.
I have a hive-partitioned dataset in a Google Cloud Storage bucket. Its size is around 54MB according to `gsutil du` and verified by downloading it and checking locally. However, if I open it with `ds = pyarrow.dataset.dataset("gs://...", partitioning="hive", format="parquet")` and then traverse it with `ds.to_batches()`, it results in multiple GB of inbound network traffic and takes much longer than simply downloading the data.
This is with PyArrow 10.0.1.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] djouallah commented on issue #33624: [GCS] Scanning hive-partitioned dataset results in orders of magnitude more network traffic than it should
Posted by GitBox <gi...@apache.org>.
djouallah commented on issue #33624:
URL: https://github.com/apache/arrow/issues/33624#issuecomment-1380408983
I am having the same issue when used with delta table
https://github.com/delta-io/delta-rs/issues/931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org