You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2022/05/12 08:57:00 UTC
[jira] [Updated] (ARROW-16530) [Go] Serial read operations on columns, even when parallel = true
[ https://issues.apache.org/jira/browse/ARROW-16530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joris Van den Bossche updated ARROW-16530:
------------------------------------------
Summary: [Go] Serial read operations on columns, even when parallel = true (was: Serial read operations on columns, even when parallel = true)
> [Go] Serial read operations on columns, even when parallel = true
> -----------------------------------------------------------------
>
> Key: ARROW-16530
> URL: https://issues.apache.org/jira/browse/ARROW-16530
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Go
> Affects Versions: 8.0.0
> Environment: Linux, golang 1.18, AMD64
> Reporter: Robert
> Priority: Major
> Labels: pull-request-available
> Fix For: 9.0.0
>
> Original Estimate: 24h
> Time Spent: 20m
> Remaining Estimate: 23h 40m
>
> I have submitted a pull request with the changes.
> https://github.com/apache/arrow/pull/13120#issuecomment-1123982147
> In pqarrow, when getting column readers for columns and struct members, the default behavior is a for loop that serially processes each column. The process of "getting" readers causes a read request, therefore causing these reads always to be issued serially. Additionally, the logic for getting next batch of records is executed in the same way, a for loop iterating through the columns. The performance impact is especially large on high-latency files such as cloud storage.
> Additionally, the code to retrieve the next batch of records also issues reads serially.
> I'm working with complex parquet files with 500+ "root" columns where some fields are lists of structs. Some of these structs have 100's of columns. In my tests, 800+ read operations are being issued to GCS serially which makes the current state of pqarrow too slow to be usable.
> The revision is to concurrently process the columns when retrieving child readers and column readers and to concurrently issue batch requests.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)