You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "LucaMaurelliC (via GitHub)" <gi...@apache.org> on 2023/11/30 14:05:08 UTC
[I] [Documentation][Parquet] Reading parquet and memory mapping [arrow]
LucaMaurelliC opened a new issue, #39005:
URL: https://github.com/apache/arrow/issues/39005
### Describe the enhancement requested
I'm learning how to exploit the arrow library in Python to read .parquet files concurrently from a local file system or a cloud file system and reading about memory mapping [here](https://arrow.apache.org/docs/python/parquet.html#reading-parquet-and-memory-mapping) I didn't understand whether I should enable it or not, and why. What are the pros and cons.
Would you mind elaborating more on this points so that users are able to evaluate and decide if that is important?
### Component(s)
Documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Documentation][Parquet] Reading parquet and memory mapping [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #39005:
URL: https://github.com/apache/arrow/issues/39005#issuecomment-3921983329
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Documentation][Parquet] Reading parquet and memory mapping [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #39005:
URL: https://github.com/apache/arrow/issues/39005#issuecomment-4620250326
This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Documentation][Parquet] Reading parquet and memory mapping [arrow]
Posted by "LucaMaurelliC (via GitHub)" <gi...@apache.org>.
LucaMaurelliC commented on issue #39005:
URL: https://github.com/apache/arrow/issues/39005#issuecomment-1835577595
What do you mean by collapsed IO?
Also, does ffspec and its implemented fs [Azure Blob Storage](https://github.com/fsspec/adlfs) exploits the PreBuffering already?
From the arrow doc:
"Coalesce and issue file reads in parallel to improve performance on high-latency filesystems (e.g. S3). If True, Arrow will use a background I/O thread pool. This option is only supported for use_legacy_dataset=False. If using a filesystem layer that itself performs readahead (e.g. fsspec’s S3FS), disable readahead for best results."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Documentation][Parquet] Reading parquet and memory mapping [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #39005:
URL: https://github.com/apache/arrow/issues/39005#issuecomment-1833912072
Memory mapping is used for local filesystem, it will use "named" `mmap` instead of directly call `readAt` on file
For file in cloud, maybe enable `PreBuffer` could help. It will merge collapsed IO and send IO by large chunk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Documentation][Parquet] Reading parquet and memory mapping [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #39005:
URL: https://github.com/apache/arrow/issues/39005#issuecomment-1835592735
Yeah, arrow has internal "pre_buffer` config, enabling it will making read-parquet issue all neccessary IO and buffer them in memory
During read a local file, arrow might just call `ReadAt` to read the local parquet small page, because it regard local read as a lightweight operation. The same straitegy might causing lots of `Get` calls for cloud storage. So it will try to "collapse" the read request: it will merge adjacent together to avoid fragment read calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Documentation][Parquet] Reading parquet and memory mapping [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #39005: [Documentation][Parquet] Reading parquet and memory mapping
URL: https://github.com/apache/arrow/issues/39005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org