You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2025/09/10 10:47:03 UTC
[Impala-ASF-CR] IMPALA-9874: Skip IO for late materialized columns
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/23012 )
Change subject: IMPALA-9874: Skip IO for late materialized columns
......................................................................
Patch Set 13:
(5 comments)
Thank you for working on this!
http://gerrit.cloudera.org:8080/#/c/23012/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23012/13//COMMIT_MSG@23
PS13, Line 23: As a result, IO bound queries with low selectivity can run
: significantly faster.
Can queries with high selectivity regress? E.g. because IO scheduling is delayed for them?
http://gerrit.cloudera.org:8080/#/c/23012/13//COMMIT_MSG@25
PS13, Line 25:
Could you please share some numbers? Did you run perf AB tests on tpch/tpcds?
http://gerrit.cloudera.org:8080/#/c/23012/13/be/src/exec/orc/hdfs-orc-scanner.cc
File be/src/exec/orc/hdfs-orc-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/23012/13/be/src/exec/orc/hdfs-orc-scanner.cc@926
PS13, Line 926: rows_selected_in_row_group_ = false;
Does it do anything for ORC?
http://gerrit.cloudera.org:8080/#/c/23012/13/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/23012/13/be/src/exec/parquet/hdfs-parquet-scanner.cc@259
PS13, Line 259: late_materialization_ = !(
: // Late materialization will be disabled if
: filter_readers_.empty() // no columns are used for filtering, or
: || non_filter_readers_.empty() // all columns are used for filtering, or
: || late_materialization_threshold_
: < 0 // the threshold is set to negative in query options, or
: || filter_readers_[0]->max_rep_level()
: > 0 // the first filter column belongs to a repeated field, or
: || HasStructColumnReader(
: non_filter_readers_) // a non-filter reader has struct readers.
nit: could be moved to its own method
Btw, thanks for adding the comments, it makes it much easier to understand!
http://gerrit.cloudera.org:8080/#/c/23012/13/be/src/exec/parquet/hdfs-parquet-scanner.cc@1051
PS13, Line 1051: RETURN_IF_ERROR(
: BaseScalarColumnReader::StartScans(non_dict_filterable_filter_readers_));
: status =
: BaseScalarColumnReader::InitDictionaries(non_dict_filterable_filter_readers_);
We can see this pattern 3X in this file, I think it's worth to create a function for this.
--
To view, visit http://gerrit.cloudera.org:8080/23012
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4a052b028220517503e634e3f916d1fbd60eb65d
Gerrit-Change-Number: 23012
Gerrit-PatchSet: 13
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Sep 2025 10:47:03 +0000
Gerrit-HasComments: Yes