You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Weston Pace (Jira)" <ji...@apache.org> on 2021/01/15 19:43:00 UTC
[jira] [Created] (ARROW-11262) Move decompression off background
reader thread into thread pool
Weston Pace created ARROW-11262:
-----------------------------------
Summary: Move decompression off background reader thread into thread pool
Key: ARROW-11262
URL: https://issues.apache.org/jira/browse/ARROW-11262
Project: Apache Arrow
Issue Type: Improvement
Reporter: Weston Pace
Attachments: read_csv.cc, zip-speedup.png
When reading a decompressed stream there is a fairly decent amount of CPU time spent decompressing that stream. While we are doing this we could be fetching the next block. However, the current implementation has the reading and decompressing on the same background reader thread so the next block will not be fetched until the prior block is finished decompressing.
There is still "some" ordering here, it isn't a fan-out, decompression of the blocks has to happen in sequence, but there is some gain to be had.
I created a simple example with gzip here ([https://github.com/westonpace/arrow/tree/feature/async-compressed-csv)] and you could test it with the attached example program.
On my system, when reading a 250MB gzipped CSV file there is roughly a 5% speedup if the file is cached in the OS (6.3s -> 6.0s) and a 10% to 15% speedup if the file is not cached in the OS. (~6.8s -> 6.0s)
The example requires changing the table reader implementation to receive an async generator. I think, in practice, we will want to change it to take an async input stream instead. So this may need to wait until/if we decide to expand the async paradigm into the I/O interfaces.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)