You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@arrow.apache.org by Alan Souza via user <us...@arrow.apache.org> on 2025/07/03 22:03:43 UTC

Fw: slow reading of a single parquet file when using dataset API


Yahoo Mail: Search, organise, conquer 
 
   ----- Forwarded message ----- From: "Alan Souza" <al...@rocketmail.com> To: "Dev" <de...@arrow.apache.org> Cc:  Sent: Thu, 3 Jul 2025 at 18:09 Subject: slow reading of a single parquet file when using dataset API  Hi, I am testing the dataset API from the Arrow cpp to scan a single file, to use the predicate pushdown and project facilities of this API. However the performance appears to be very slow, spending most of the time with pagefaults/allocations.
The Parquet that I am reading contains some dozens of columns, but I only use one for predicate pushdown and one to read (project). The column that I am reading is relatively big consisting of a large list float with one to two thousand float samples per row. I am selecting about a  million rows from a file with approximately 2 millions rows (7GB Parquet file)
I got this flamegraph
    

for the following code:

 Any suggestions on how to improve the performance ?thanks