You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "bwi-earth (via GitHub)" <gi...@apache.org> on 2025/03/18 20:54:18 UTC

[I] [Python] Order of reading pyarrow.dataset.Dataset [arrow]

bwi-earth opened a new issue, #45855:
URL: https://github.com/apache/arrow/issues/45855

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   I'm reading about various methods to consume a `pyarrow.dataset.Dataset`, in the of large dataset (.to_table is excluded).
   
   it seems that it is impossible to read a dataset in by chunk, yet in ordered manner, `to_batches` doesn't offer any guarntees about the order of the retrieved batches.
   
   The best I've come up with is to list the fragments of the dataset and read each one individually, then sort partial outputs.
   
   However, if that's the case i'm loosing the benefit of pyarrow loading stuff in the background.
   
   (im using parquet stored in s3 as backend, doesn't seem to be relevant though)
   
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Order of reading pyarrow.dataset.Dataset [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #45855:
URL: https://github.com/apache/arrow/issues/45855#issuecomment-4089449785

   This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this usage question has evolved into a feature request or docs update, please remove the 'Type: usage' label and add the 'Type: enhancement' label instead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Order of reading pyarrow.dataset.Dataset [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #45855: [Python] Order of reading pyarrow.dataset.Dataset
URL: https://github.com/apache/arrow/issues/45855


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org