[jira] [Comment Edited] (ARROW-12428) [Python] pyarrow.parquet.read_* should use pre_buffer=True

["David Li (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-12428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324043#comment-17324043 ] 

David Li edited comment on ARROW-12428 at 4/16/21, 7:41 PM:
------------------------------------------------------------

And for local files, to confirm that pre_buffer isn't a negative:
{noformat}
Pandas: 14.584974920144305 seconds
PyArrow: 6.650648137088865 seconds
PyArrow (pre-buffer): 6.587288308190182 seconds
{noformat}
This is on a system with NVME storage, so results may vary for spinning-rust or SATA SSDs.

(Updated results to read once without measuring before taking the measurement, in case disk cache is a factor)


was (Author: lidavidm):
And for local files, to confirm that pre_buffer isn't a negative:
{noformat}
Pandas: 14.566267257090658 seconds
PyArrow: 6.649410092970356 seconds
PyArrow (pre-buffer): 6.627140663098544 seconds {noformat}
This is on a system with NVME storage, so results may vary for spinning-rust or SATA SSDs.

> [Python] pyarrow.parquet.read_* should use pre_buffer=True
> ----------------------------------------------------------
>
>                 Key: ARROW-12428
>                 URL: https://issues.apache.org/jira/browse/ARROW-12428
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>            Reporter: David Li
>            Assignee: David Li
>            Priority: Major
>             Fix For: 5.0.0
>
>
> If the user is synchronously reading a single file, we should try to read it as fast as possible. The one sticking point might be whether it's beneficial to enable this no matter the filesystem or whether we should try to only enable it on high-latency filesystems.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)