You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Andrew Lamb (Jira)" <ji...@apache.org> on 2020/08/09 10:26:00 UTC
[jira] [Commented] (ARROW-9674) [Rust] Parquet reader should
implement Send + Sync
[ https://issues.apache.org/jira/browse/ARROW-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173800#comment-17173800 ]
Andrew Lamb commented on ARROW-9674:
------------------------------------
Doing this would also make multi-threading parquet processing easier. I have thought about this, and one thing I was thinking about was that the parquet code now uses `Rc` to pass around the various metadata structures.
Setting up the code to work well with async/multi-threaded might benefit from changing uses of `Rc` to `Arc` throughout the metadata handling code; That change, however, would likely be a breaking API change
> [Rust] Parquet reader should implement Send + Sync
> --------------------------------------------------
>
> Key: ARROW-9674
> URL: https://issues.apache.org/jira/browse/ARROW-9674
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Rust
> Reporter: Andy Grove
> Priority: Major
>
> To use the parquet reader in async code it is necessary to run it on its own thread and then use channels to communicate and fetch batches. This is very inefficient and it would be better if the main Parquet traits implemented Sync + Send.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)