You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Balazs Hevele (Code Review)" <ge...@cloudera.org> on 2026/03/03 12:54:11 UTC
[Impala-ASF-CR] WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Balazs Hevele has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24063
Change subject: WIP IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-common.h
M be/src/runtime/string-value.h
2 files changed, 5 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/1
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 1
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: WIP IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21844/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 1
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2026 13:19:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 10: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13231/
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 10
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 15:55:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21975/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 15
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Mar 2026 09:34:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#14).
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
IMPALA-14794: Implement small string optimization in Parquet scanner
When decoding a string in a parquet file, it is now always smallified
if possible.
With plain encoding, if a column's data page only contains smallified
strings, the data page's memory is no longer attached to the tuple,
since no string points into it.
With dictionary encoding, string column readers always allocate a copy
of the dictionary page so strings can point into it. If all strings are
smallified, this copy is freed after decoding all data, because no
strings point into it.
Measurements:
Measured time of a join query where lots of small strings are sent from
the reader:
select l1.* from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 84.385ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.183ms
After:
KrpcDataStreamSender: SerializeBatchTime: 67.598ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.632ms
Same measurement with a table with a higher scale factor:
select l1.* from tpch30_parquet_snap.lineitem l1
join tpch30_parquet_snap.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 2s359ms
HDFS_SCAN_NODE: MaterializeTupleTime: 239.267ms
After:
KrpcDataStreamSender: SerializeBatchTime: 1s702ms
HDFS_SCAN_NODE: MaterializeTupleTime: 243.606ms
This is ~27% gain in SerializeBatchTime.
Testing:
-Added a test to parquet-plain-test.cc to test that small strings are
smallified upon decoding
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-plain-test.cc
M be/src/runtime/smallable-string.h
M be/src/runtime/string-value.h
M be/src/testutil/random-vector-generators.h
8 files changed, 186 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/14
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 14
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 10: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 10
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 14:29:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#12).
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
IMPALA-14794: Implement small string optimization in Parquet scanner
When decoding a string in a parquet file, it is now always smallified
if possible.
With plain encoding, if a column's data page only contains smallified
strings, the data page's memory is no longer attached to the tuple,
since no string points into it.
With dictionary encoding, string column readers always allocate a copy
of the dictionary page so strings can point into it. If all strings are
smallified, this copy is freed after decoding all data, because no
strings point into it.
Measurements:
Measured time of a join query where lots of small strings are sent from
the reader:
select l1.* from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 84.385ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.183ms
After:
KrpcDataStreamSender: SerializeBatchTime: 67.598ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.632ms
Same measurement with a table with a higher scale factor:
select l1.* from tpch30_parquet_snap.lineitem l1
join tpch30_parquet_snap.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 2s359ms
HDFS_SCAN_NODE: MaterializeTupleTime: 239.267ms
After:
KrpcDataStreamSender: SerializeBatchTime: 1s702ms
HDFS_SCAN_NODE: MaterializeTupleTime: 243.606ms
This is ~27% gain in SerializeBatchTime.
Testing:
-Added a test to parquet-plain-test.cc to test that small strings are
smallified upon decoding
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-plain-test.cc
M be/src/runtime/smallable-string.h
M be/src/runtime/string-value.h
M be/src/testutil/random-vector-generators.h
8 files changed, 182 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/12
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 12
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 9:
There was a crash found in https://jenkins.impala.io/job/perf-AB-test-ub2004/390
dict_page_pool_ can be nullptr when there are no var len values. In this case, there is no need to call FreeAll/AcquireData in CreateDictionaryDecoder, so added a nullptr check around it.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 9
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2026 13:22:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 11:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/21953/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 11
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Mar 2026 07:40:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#2).
Change subject: WIP IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
WIP IMPALA-14794: Implement small string optimization in Parquet scanner
TODO:
-do not copy the IO buffer for output batches if there are only small
strings
-measure performance (can measure with small strings in dictionary, or
small strings without dictionary)
-test?
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-common.h
M be/src/runtime/string-value.h
2 files changed, 5 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/2
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 2
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#6).
Change subject: WIP IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
WIP IMPALA-14794: Implement small string optimization in Parquet scanner
When decoding a string in a parquet files, it is now always smallified
if possible.
With plain encoding, if a column's data page only contains smallified
strings, the data page's memory is no longer attached to the tuple,
since no string points into it.
With dictionary encoding, string column readers always allocate a copy
of the dictionary page so strings can point into it. If all strings are
smallified, this copy is freed after decoding all data, because no
strings point into it.
Measurements:
Measured time of a join query where lots of small strings are sent from
the reader:
select l1.* from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: ~90.0ms
After:
KrpcDataStreamSender: SerializeBatchTime: ~65.0ms
This is ~28% gain in speed. Most of the gain comes from deepcopy.
TODO:
-test?
-test that all cases work properly (plain/dict encoding,
small/large strings)
-test that page data is not actually attached to the tuple when
there are only small strings?
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/runtime/string-value.h
5 files changed, 127 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/6
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 6
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13243/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 15
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Mar 2026 10:12:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 17: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 17
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 Mar 2026 17:47:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24063/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24063/7//COMMIT_MSG@20
PS7, Line 20: Measured time of a join query where lots of small strings are sent from
: the reader:
: select l1.* from tpch_parquet.lineitem l1
: join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
: limit 1;
> Can you also run one of the following jobs?
The jobs run are:
https://jenkins.impala.io/job/perf-AB-test-ub2004/389/
https://jenkins.impala.io/job/perf-AB-test-ub2004/391/
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 9
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2026 19:19:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#8).
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
IMPALA-14794: Implement small string optimization in Parquet scanner
When decoding a string in a parquet file, it is now always smallified
if possible.
With plain encoding, if a column's data page only contains smallified
strings, the data page's memory is no longer attached to the tuple,
since no string points into it.
With dictionary encoding, string column readers always allocate a copy
of the dictionary page so strings can point into it. If all strings are
smallified, this copy is freed after decoding all data, because no
strings point into it.
Measurements:
Measured time of a join query where lots of small strings are sent from
the reader:
select l1.* from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 84.385ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.183ms
After:
KrpcDataStreamSender: SerializeBatchTime: 67.598ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.632ms
Same measurement with a table with a higher scale factor:
select l1.* from tpch30_parquet_snap.lineitem l1
join tpch30_parquet_snap.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 2s359ms
HDFS_SCAN_NODE: MaterializeTupleTime: 239.267ms
After:
KrpcDataStreamSender: SerializeBatchTime: 1s702ms
HDFS_SCAN_NODE: MaterializeTupleTime: 243.606ms
This is ~27% gain in SerializeBatchTime.
Testing:
-Added a test to parquet-plain-test.cc to test that small strings are
smallified upon decoding
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-plain-test.cc
M be/src/runtime/smallable-string.h
M be/src/runtime/string-value.h
M be/src/testutil/random-vector-generators.h
8 files changed, 178 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/8
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 8
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 13: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 13
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Mar 2026 13:42:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 13: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13239/
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 13
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Mar 2026 17:57:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 17:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13248/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 17
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 Mar 2026 13:24:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 17: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 17
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 Mar 2026 13:24:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#7).
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
IMPALA-14794: Implement small string optimization in Parquet scanner
When decoding a string in a parquet file, it is now always smallified
if possible.
With plain encoding, if a column's data page only contains smallified
strings, the data page's memory is no longer attached to the tuple,
since no string points into it.
With dictionary encoding, string column readers always allocate a copy
of the dictionary page so strings can point into it. If all strings are
smallified, this copy is freed after decoding all data, because no
strings point into it.
Measurements:
Measured time of a join query where lots of small strings are sent from
the reader:
select l1.* from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: ~90.0ms
After:
KrpcDataStreamSender: SerializeBatchTime: ~65.0ms
This is ~28% gain in speed.
Testing:
-Added a test to parquet-plain-test.cc to test that small strings are
smallified upon decoding
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-plain-test.cc
M be/src/runtime/string-value.h
M be/src/testutil/random-vector-generators.h
7 files changed, 168 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/7
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 7
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: WIP IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21865/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 4
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 12:29:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#5).
Change subject: WIP IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
WIP IMPALA-14794: Implement small string optimization in Parquet scanner
TODO:
-measure performance (can measure with small strings in dictionary, or
small strings without dictionary)
-test?
-test that all cases work properly (plain/dict encoding,
small/large strings)
-test that page data is not actually attached to the tuple when
there are only small strings?
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/runtime/string-value.h
5 files changed, 127 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/5
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 5
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13231/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 10
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 14:29:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21918/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 9
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2026 13:49:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: WIP IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21867/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 5
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 14:19:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21965/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 11
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Mar 2026 08:35:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 13:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13239/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 13
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Mar 2026 13:42:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#3).
Change subject: WIP IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
WIP IMPALA-14794: Implement small string optimization in Parquet scanner
TODO:
-do not copy the IO buffer for output batches if there are only small
strings
-measure performance (can measure with small strings in dictionary, or
small strings without dictionary)
-test?
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-common.h
M be/src/runtime/string-value.h
4 files changed, 14 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/3
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 3
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 7:
(7 comments)
Thanks for working on this!
http://gerrit.cloudera.org:8080/#/c/24063/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24063/7//COMMIT_MSG@20
PS7, Line 20: Measured time of a join query where lots of small strings are sent from
: the reader:
: select l1.* from tpch_parquet.lineitem l1
: join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
: limit 1;
Can you also run one of the following jobs?
* https://jenkins.impala.io/job/perf-AB-test-ub2204/
* https://jenkins.impala.io/job/perf-AB-test-ub2004/
Both TPC-H and TPC-DS can be interesting.
http://gerrit.cloudera.org:8080/#/c/24063/7//COMMIT_MSG@26
PS7, Line 26: Before:
: KrpcDataStreamSender: SerializeBatchTime: ~90.0ms
: After:
: KrpcDataStreamSender: SerializeBatchTime: ~65.0ms
Can you include other metrics, e.g. scan times (e.g. MaterializeTupleTime)? Also, for better stability, you could user a higher scale factor.
You could load it via
bin/load-data.py -s 30 -f --workloads tpch --table_formats text/none,parquet/snap
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-chunk-reader.h
File be/src/exec/parquet/parquet-column-chunk-reader.h:
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-chunk-reader.h@176
PS7, Line 176: by the row batch.
nit: by the last row batch.
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-chunk-reader.cc
File be/src/exec/parquet/parquet-column-chunk-reader.cc:
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-chunk-reader.cc@173
PS7, Line 173: the row batch.
nit: the last row batch.
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-readers.cc
File be/src/exec/parquet/parquet-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-readers.cc@837
PS7, Line 837: all_strings_smallified = false;
We could add DCHECK_FALSE(val.Smallify());
To verify we didn't miss the opportunity to smallify the string.
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-readers.cc@870
PS7, Line 870: col_chunk_reader_.keep_data_page_pool_ = true;
We could add DCHECK_FALSE(val.Smallify());
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-readers.cc@963
PS7, Line 963: if (!val->IsSmall()) {
We could add DCHECK_FALSE(val.Smallify());
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 7
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2026 17:40:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 16: Code-Review+2
LGTM!
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 16
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 Mar 2026 13:24:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#16).
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
IMPALA-14794: Implement small string optimization in Parquet scanner
When decoding a string in a parquet file, it is now always smallified
if possible.
With plain encoding, if a column's data page only contains smallified
strings, the data page's memory is no longer attached to the tuple,
since no string points into it.
With dictionary encoding, string column readers always allocate a copy
of the dictionary page so strings can point into it. If all strings are
smallified, this copy is freed after decoding all data, because no
strings point into it.
Measurements:
Measured time of a join query where lots of small strings are sent from
the reader:
select l1.* from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 84.385ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.183ms
After:
KrpcDataStreamSender: SerializeBatchTime: 67.598ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.632ms
Same measurement with a table with a higher scale factor:
select l1.* from tpch30_parquet_snap.lineitem l1
join tpch30_parquet_snap.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 2s359ms
HDFS_SCAN_NODE: MaterializeTupleTime: 239.267ms
After:
KrpcDataStreamSender: SerializeBatchTime: 1s702ms
HDFS_SCAN_NODE: MaterializeTupleTime: 243.606ms
This is ~27% gain in SerializeBatchTime.
Testing:
-Added a test to parquet-plain-test.cc to test that small strings are
smallified upon decoding
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-plain-test.cc
M be/src/runtime/smallable-string.h
M be/src/runtime/string-value.h
M be/src/testutil/random-vector-generators.h
8 files changed, 190 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/16
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 16
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21976/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 16
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Mar 2026 11:38:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 11: Code-Review+1
(5 comments)
Found a few formatting issues, otherwise LGTM!
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc
File be/src/exec/parquet/parquet-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc@861
PS11, Line 861: S
nit: missing space
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc@864
PS11, Line 864:
nit: requires +4 indent
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc@955
PS11, Line 955:
nit: requires +4 indent
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc@956
PS11, Line 956:
nit: requires +4 indent
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc@967
PS11, Line 967:
nit: requires +4 indent
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 11
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Mar 2026 10:26:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 12: Code-Review+2
LGTM!
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 12
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Mar 2026 13:41:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 16:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/exec/parquet/parquet-column-chunk-reader.cc
File be/src/exec/parquet/parquet-column-chunk-reader.cc:
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/exec/parquet/parquet-column-chunk-reader.cc@84
PS15, Line 84: if (mem_pool != nullptr) {
> If mem_pool is not nullptr, it's probably safer to pass ownership to it via
Done
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/exec/parquet/parquet-column-readers.cc
File be/src/exec/parquet/parquet-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/exec/parquet/parquet-column-readers.cc@848
PS15, Line 848:
> nit: requires +4 indent
Done
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/runtime/smallable-string.h
File be/src/runtime/smallable-string.h:
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/runtime/smallable-string.h@125
PS15, Line 125: r
> nit: parenthesis are redundant
Done
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 16
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Mar 2026 11:11:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 8:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/24063/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24063/7//COMMIT_MSG@26
PS7, Line 26: Before:
: KrpcDataStreamSender: SerializeBatchTime: 84.385ms
: HDFS_SCAN_NODE: MaterializeTupleTime: 8.183ms
: After:
> Can you include other metrics, e.g. scan times (e.g. MaterializeTupleTime)?
Done
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-chunk-reader.h
File be/src/exec/parquet/parquet-column-chunk-reader.h:
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-chunk-reader.h@176
PS7, Line 176: by the last row b
> nit: by the last row batch.
Done
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-chunk-reader.cc
File be/src/exec/parquet/parquet-column-chunk-reader.cc:
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-chunk-reader.cc@173
PS7, Line 173: the last row b
> nit: the last row batch.
Done
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-readers.cc
File be/src/exec/parquet/parquet-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-readers.cc@837
PS7, Line 837: DCHECK(!val.CanBeSmallified());
> We could add DCHECK_FALSE(val.Smallify());
Done
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-readers.cc@870
PS7, Line 870: if (!val->IsSmall()) {
> We could add DCHECK_FALSE(val.Smallify());
Done
http://gerrit.cloudera.org:8080/#/c/24063/7/be/src/exec/parquet/parquet-column-readers.cc@963
PS7, Line 963: for (int i = 0; i < count; ++i) {
> We could add DCHECK_FALSE(val.Smallify());
Done
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 8
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2026 09:18:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#4).
Change subject: WIP IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
WIP IMPALA-14794: Implement small string optimization in Parquet scanner
TODO:
-do not copy the IO buffer for output batches if there are only small
strings (Dictionary encoded case is remaining)
-measure performance (can measure with small strings in dictionary, or
small strings without dictionary)
-test?
-test that page data is not actually attached to the tuple when
there are only small strings?
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/runtime/string-value.h
5 files changed, 80 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/4
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 4
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 11:
Fixed a bug with using wrong stride
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 11
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Mar 2026 07:10:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21893/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 7
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Mar 2026 11:53:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 15: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 15
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Mar 2026 14:30:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21969/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 12
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Mar 2026 13:58:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 16: Code-Review-1
Included changes from somewhere else by mistake
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 16
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Mar 2026 11:11:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 12:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc
File be/src/exec/parquet/parquet-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc@861
PS11, Line 861:
> nit: missing space
Done
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc@864
PS11, Line 864:
> nit: requires +4 indent
Done
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc@955
PS11, Line 955:
> nit: requires +4 indent
Done
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc@956
PS11, Line 956:
> nit: requires +4 indent
Done
http://gerrit.cloudera.org:8080/#/c/24063/11/be/src/exec/parquet/parquet-column-readers.cc@967
PS11, Line 967:
> nit: requires +4 indent
Done
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 12
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Mar 2026 13:36:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21915/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 8
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2026 09:44:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#9).
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
IMPALA-14794: Implement small string optimization in Parquet scanner
When decoding a string in a parquet file, it is now always smallified
if possible.
With plain encoding, if a column's data page only contains smallified
strings, the data page's memory is no longer attached to the tuple,
since no string points into it.
With dictionary encoding, string column readers always allocate a copy
of the dictionary page so strings can point into it. If all strings are
smallified, this copy is freed after decoding all data, because no
strings point into it.
Measurements:
Measured time of a join query where lots of small strings are sent from
the reader:
select l1.* from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 84.385ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.183ms
After:
KrpcDataStreamSender: SerializeBatchTime: 67.598ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.632ms
Same measurement with a table with a higher scale factor:
select l1.* from tpch30_parquet_snap.lineitem l1
join tpch30_parquet_snap.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 2s359ms
HDFS_SCAN_NODE: MaterializeTupleTime: 239.267ms
After:
KrpcDataStreamSender: SerializeBatchTime: 1s702ms
HDFS_SCAN_NODE: MaterializeTupleTime: 243.606ms
This is ~27% gain in SerializeBatchTime.
Testing:
-Added a test to parquet-plain-test.cc to test that small strings are
smallified upon decoding
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-plain-test.cc
M be/src/runtime/smallable-string.h
M be/src/runtime/string-value.h
M be/src/testutil/random-vector-generators.h
8 files changed, 180 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/9
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 9
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#11).
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
IMPALA-14794: Implement small string optimization in Parquet scanner
When decoding a string in a parquet file, it is now always smallified
if possible.
With plain encoding, if a column's data page only contains smallified
strings, the data page's memory is no longer attached to the tuple,
since no string points into it.
With dictionary encoding, string column readers always allocate a copy
of the dictionary page so strings can point into it. If all strings are
smallified, this copy is freed after decoding all data, because no
strings point into it.
Measurements:
Measured time of a join query where lots of small strings are sent from
the reader:
select l1.* from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 84.385ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.183ms
After:
KrpcDataStreamSender: SerializeBatchTime: 67.598ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.632ms
Same measurement with a table with a higher scale factor:
select l1.* from tpch30_parquet_snap.lineitem l1
join tpch30_parquet_snap.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 2s359ms
HDFS_SCAN_NODE: MaterializeTupleTime: 239.267ms
After:
KrpcDataStreamSender: SerializeBatchTime: 1s702ms
HDFS_SCAN_NODE: MaterializeTupleTime: 243.606ms
This is ~27% gain in SerializeBatchTime.
Testing:
-Added a test to parquet-plain-test.cc to test that small strings are
smallified upon decoding
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-plain-test.cc
M be/src/runtime/smallable-string.h
M be/src/runtime/string-value.h
M be/src/testutil/random-vector-generators.h
8 files changed, 182 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/11
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 11
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24063
to look at the new patch set (#15).
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
IMPALA-14794: Implement small string optimization in Parquet scanner
When decoding a string in a parquet file, it is now always smallified
if possible.
With plain encoding, if a column's data page only contains smallified
strings, the data page's memory is no longer attached to the tuple,
since no string points into it.
With dictionary encoding, string column readers always allocate a copy
of the dictionary page so strings can point into it. If all strings are
smallified, this copy is freed after decoding all data, because no
strings point into it.
Measurements:
Measured time of a join query where lots of small strings are sent from
the reader:
select l1.* from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 84.385ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.183ms
After:
KrpcDataStreamSender: SerializeBatchTime: 67.598ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.632ms
Same measurement with a table with a higher scale factor:
select l1.* from tpch30_parquet_snap.lineitem l1
join tpch30_parquet_snap.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 2s359ms
HDFS_SCAN_NODE: MaterializeTupleTime: 239.267ms
After:
KrpcDataStreamSender: SerializeBatchTime: 1s702ms
HDFS_SCAN_NODE: MaterializeTupleTime: 243.606ms
This is ~27% gain in SerializeBatchTime.
Testing:
-Added a test to parquet-plain-test.cc to test that small strings are
smallified upon decoding
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-plain-test.cc
M be/src/runtime/smallable-string.h
M be/src/runtime/string-value.h
M be/src/testutil/random-vector-generators.h
8 files changed, 186 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/24063/15
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 15
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 15:
There was a crash when trying to read a corrupted dictionary page, because the page copy in ParquetColumnChunkReader::dict_page_pool_ was not freed after the error. Now it is freed in ParquetColumnChunkReader::Close() in this case.
Everything in test_scanners.py::TestParquet succeeded locally.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 15
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Mar 2026 09:07:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21974/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 14
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Mar 2026 09:22:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 15:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/exec/parquet/parquet-column-chunk-reader.cc
File be/src/exec/parquet/parquet-column-chunk-reader.cc:
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/exec/parquet/parquet-column-chunk-reader.cc@84
PS15, Line 84: dict_page_pool_->FreeAll();
If mem_pool is not nullptr, it's probably safer to pass ownership to it via 'mem_pool->AcquireData()'.
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/exec/parquet/parquet-column-readers.cc
File be/src/exec/parquet/parquet-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/exec/parquet/parquet-column-readers.cc@848
PS15, Line 848:
nit: requires +4 indent
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/runtime/smallable-string.h
File be/src/runtime/smallable-string.h:
http://gerrit.cloudera.org:8080/#/c/24063/15/be/src/runtime/smallable-string.h@125
PS15, Line 125: (
nit: parenthesis are redundant
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 15
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Mar 2026 10:12:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 16: -Code-Review
Never mind last comment.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 16
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Mar 2026 11:13:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 9: Code-Review+2
LGTM!
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 9
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 14:29:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
IMPALA-14794: Implement small string optimization in Parquet scanner
When decoding a string in a parquet file, it is now always smallified
if possible.
With plain encoding, if a column's data page only contains smallified
strings, the data page's memory is no longer attached to the tuple,
since no string points into it.
With dictionary encoding, string column readers always allocate a copy
of the dictionary page so strings can point into it. If all strings are
smallified, this copy is freed after decoding all data, because no
strings point into it.
Measurements:
Measured time of a join query where lots of small strings are sent from
the reader:
select l1.* from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 84.385ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.183ms
After:
KrpcDataStreamSender: SerializeBatchTime: 67.598ms
HDFS_SCAN_NODE: MaterializeTupleTime: 8.632ms
Same measurement with a table with a higher scale factor:
select l1.* from tpch30_parquet_snap.lineitem l1
join tpch30_parquet_snap.lineitem l2 on l1.l_shipmode = l2.l_shipmode
limit 1;
Before:
KrpcDataStreamSender: SerializeBatchTime: 2s359ms
HDFS_SCAN_NODE: MaterializeTupleTime: 239.267ms
After:
KrpcDataStreamSender: SerializeBatchTime: 1s702ms
HDFS_SCAN_NODE: MaterializeTupleTime: 243.606ms
This is ~27% gain in SerializeBatchTime.
Testing:
-Added a test to parquet-plain-test.cc to test that small strings are
smallified upon decoding
Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Reviewed-on: http://gerrit.cloudera.org:8080/24063
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/parquet/parquet-column-chunk-reader.cc
M be/src/exec/parquet/parquet-column-chunk-reader.h
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-plain-test.cc
M be/src/runtime/smallable-string.h
M be/src/runtime/string-value.h
M be/src/testutil/random-vector-generators.h
8 files changed, 190 insertions(+), 6 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 18
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14794: Implement small string optimization in Parquet scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24063 )
Change subject: WIP IMPALA-14794: Implement small string optimization in Parquet scanner
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21863/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c550d35cd6d3ec259b899b325611294137ccef
Gerrit-Change-Number: 24063
Gerrit-PatchSet: 3
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 07:37:21 +0000
Gerrit-HasComments: No