You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "yaelperl1234 (via GitHub)" <gi...@apache.org> on 2023/03/14 22:18:18 UTC
[GitHub] [arrow] yaelperl1234 opened a new issue, #34566: pyarrow.parquet.read_table is very long for the first time
yaelperl1234 opened a new issue, #34566:
URL: https://github.com/apache/arrow/issues/34566
### Describe the usage question you have. Please include as many useful details as possible.
I use pyarrow.parquet.read_table to read a parquet dataset on a local filesystem, and it takes much more time for the first read call than the next read calls.
(26 seconds vs 2 seconds on ~500 partitions).
Does it use any kind of caching? where?
And how can I reduce the time of the first read call?
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] pyarrow.parquet.read_table is very long for the first read call [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #34566:
URL: https://github.com/apache/arrow/issues/34566#issuecomment-2966751672
This issue appears to be stale or discussion appears to be concluded, so I'm going to close it for now. Please feel free to reopen if it's still relevant or you'd like to continue the discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34566: [Python] pyarrow.parquet.read_table is very long for the first read call
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34566:
URL: https://github.com/apache/arrow/issues/34566#issuecomment-1474423947
Both Linux and Windows have their own file caches. Pyarrow does not do any caching of its own.
How exactly are you reading the files? Are you looping over a list of files and calling `read_table` on each one? Or are you using some method to read an entire dataset? Is it possible the initial delay is the time it takes to list and discover all the files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yaelperl1234 commented on issue #34566: [Python] pyarrow.parquet.read_table is very long for the first read call
Posted by "yaelperl1234 (via GitHub)" <gi...@apache.org>.
yaelperl1234 commented on issue #34566:
URL: https://github.com/apache/arrow/issues/34566#issuecomment-1475375059
@westonpace
I read the entire dataset with `read_table(<path to parquet base dir>)`
Assuming the initial delay is the time it takes to list all files - do you think that saving the dataset without partitions might make it faster?
Or can I save somehow the indexes of the files in advance?
I run on: Ubuntu 18.04.4 LTS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] pyarrow.parquet.read_table is very long for the first read call [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #34566: [Python] pyarrow.parquet.read_table is very long for the first read call
URL: https://github.com/apache/arrow/issues/34566
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34566: [Python] pyarrow.parquet.read_table is very long for the first read call
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34566:
URL: https://github.com/apache/arrow/issues/34566#issuecomment-1479876277
> Assuming the initial delay is the time it takes to list all files - do you think that saving the dataset without partitions might make it faster?
It's worth benchmarking. Are these files local? Or onS3?
> Or can I save somehow the indexes of the files in advance?
If you have the list of filenames (which you can save in whatever format you want) and the dataset schema (which you can save as an empty arrow/parquet file) then you can do:
```
table = ds.dataset(list_of_paths, schema=dataset_schema).to_table()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org