You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "Jinpeng Zhou (Jira)" <ji...@apache.org> on 2024/01/27 03:08:00 UTC
[jira] [Created] (PARQUET-2423) Avoid allocating buffer obeject in RecordReader's SkipRecords
Jinpeng Zhou created PARQUET-2423:
-------------------------------------
Summary: Avoid allocating buffer obeject in RecordReader's SkipRecords
Key: PARQUET-2423
URL: https://issues.apache.org/jira/browse/PARQUET-2423
Project: Parquet
Issue Type: Improvement
Components: parquet-cpp
Reporter: Jinpeng Zhou
Currently each invocation of SkipRecords() for non-repeated fields will create a brand new buffer object[1]. I think it probably worth keep the buffer object alive and just resize it for each skip, as the buffer is just a bitmap (i.e., should remain quite small even we don't free its memory after skip). Performance results are as follows:
Keep buffer object alive:
--------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
--------------------------------------------------------------------------------------------------------------
*RecordReaderSkipRecords/Repetition:0/BatchSize:1 29958201 ns 29943377 ns 23 bytes_per_second=163.068M/s*
*RecordReaderSkipRecords/Repetition:0/BatchSize:10 3190298 ns 3190524 ns 227 bytes_per_second=1.49454G/s*
*RecordReaderSkipRecords/Repetition:0/BatchSize:100 479056 ns 480437 ns 1500 bytes_per_second=9.92507G/s*
*RecordReaderSkipRecords/Repetition:0/BatchSize:1000 256497 ns 257725 ns 2763 bytes_per_second=18.5018G/s*
RecordReaderSkipRecords/Repetition:1/BatchSize:1000 2910364 ns 2910479 ns 239 bytes_per_second=893.189M/s
RecordReaderSkipRecords/Repetition:2/BatchSize:1000 20539472 ns 20535632 ns 34 bytes_per_second=135.007M/s
Recreate upon each skip (current behavior):
--------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
--------------------------------------------------------------------------------------------------------------
*RecordReaderSkipRecords/Repetition:0/BatchSize:1 33261760 ns 33199124 ns 21 bytes_per_second=147.077M/s*
*RecordReaderSkipRecords/Repetition:0/BatchSize:10 3256993 ns 3254609 ns 216 bytes_per_second=1.46511G/s*
*RecordReaderSkipRecords/Repetition:0/BatchSize:100 492856 ns 493377 ns 1447 bytes_per_second=9.66477G/s*
*RecordReaderSkipRecords/Repetition:0/BatchSize:1000 262449 ns 263227 ns 2694 bytes_per_second=18.1151G/s*
RecordReaderSkipRecords/Repetition:1/BatchSize:1000 2996951 ns 2997148 ns 235 bytes_per_second=867.36M/s
RecordReaderSkipRecords/Repetition:2/BatchSize:1000 20864734 ns 20850593 ns 34 bytes_per_second=132.968M/s
[1]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org