[jira] [Updated] (ARROW-17481) [C++] [Python] Major performance improvements to CSV reading from S3

["Antoine Pitrou (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-17481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Antoine Pitrou updated ARROW-17481:
-----------------------------------
    Fix Version/s: 11.0.0

> [C++] [Python] Major performance improvements to CSV reading from S3
> --------------------------------------------------------------------
>
>                 Key: ARROW-17481
>                 URL: https://issues.apache.org/jira/browse/ARROW-17481
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++, Python
>            Reporter: Ziheng Wang
>            Assignee: Ziheng Wang
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 11.0.0
>
>          Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> The current dataset reader for CSV is pretty slow on EC2 reading from S3.
> EC2 instances have more than 3Gbps network bandwidth which make them on par with SSD. However reading batches from disk is more than 3x faster than reading from network. This should not happen.
> The reason why the dataset reader is not fully leveraging the network bandwidth is because reads are currently serial. We should change the reads to be parallel. Then even if the rest of the pipeline is not parallel we should get same read speed as disk.
> Note one might think that if you have many fragments fragment-level parallelism will take care of this. This is true to some extent however to_batches() is ordered. This means that if your fragments are big the fragment readahead will stop being effective after a while as the reader tries to deplete the fragments in order. The batch readahead for the CSV reader current is a serial readahead, which really should be a parallel readahead.
> After changing the network IO to be parallel, we should also change the parse and decode to be parallel. It's easy to change the parse to be parallel, a bit harder for the decode because of how the decoder operator works, so I will just tackle the parse first.
> On my test system (i3.2xlarge on EC2 reading from S3 one large CSV), these changes (parallel reading and parallel parsing) made reading 60 batches (~10GB) 4x faster. Note these changes will also make disk reading faster due to parallel parse.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)