You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2019/07/31 02:13:00 UTC
[jira] [Created] (ARROW-6076) [C++][Parquet] RecordReader::Reset
logic is inefficient for small reads
Wes McKinney created ARROW-6076:
-----------------------------------
Summary: [C++][Parquet] RecordReader::Reset logic is inefficient for small reads
Key: ARROW-6076
URL: https://issues.apache.org/jira/browse/ARROW-6076
Project: Apache Arrow
Issue Type: Improvement
Components: C++
Reporter: Wes McKinney
Fix For: 1.0.0
We have a unit test
https://github.com/apache/arrow/blob/master/cpp/src/parquet/arrow/arrow-reader-writer-test.cc#L933
that reads 1 record at a time from a Parquet-Arrow column reader. There is logic on RecordReader that advances the definition/repetition levels based on consumed data from previous records, but this is inefficient for this case:
https://github.com/apache/arrow/blob/master/cpp/src/parquet/column_reader.cc#L1011
This should be refactored to not require this copying, or at least to only "shift" the levels occasionally
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)