You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2022/05/14 07:05:21 UTC
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18528
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
CollectionItemsRead in the runtime profile counts the total number of
nested collection items read by the scan node. Only created for scans
that support nested types, e.g. Parquet or ORC.
Each scanner thread maintains its local counter and merges it into
HdfsScanNode counter for each row batch. However, the local counter in
orc-scanner is uninitialized, leading to weird values. This patch simply
initializes it to 0 and add test coverage.
Tests:
- Add profile verifiers in tpch tests. There are no nested tables so
the CollectionItemsRead should all be 0.
Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
---
M be/src/exec/hdfs-orc-scanner.h
M testdata/workloads/tpch/queries/tpch-q1.test
M testdata/workloads/tpch/queries/tpch-q10.test
M testdata/workloads/tpch/queries/tpch-q11.test
M testdata/workloads/tpch/queries/tpch-q12.test
M testdata/workloads/tpch/queries/tpch-q13.test
M testdata/workloads/tpch/queries/tpch-q14.test
M testdata/workloads/tpch/queries/tpch-q15.test
M testdata/workloads/tpch/queries/tpch-q16.test
M testdata/workloads/tpch/queries/tpch-q17.test
M testdata/workloads/tpch/queries/tpch-q18.test
M testdata/workloads/tpch/queries/tpch-q19.test
M testdata/workloads/tpch/queries/tpch-q2.test
M testdata/workloads/tpch/queries/tpch-q20.test
M testdata/workloads/tpch/queries/tpch-q21.test
M testdata/workloads/tpch/queries/tpch-q22.test
M testdata/workloads/tpch/queries/tpch-q3.test
M testdata/workloads/tpch/queries/tpch-q4.test
M testdata/workloads/tpch/queries/tpch-q5.test
M testdata/workloads/tpch/queries/tpch-q6.test
M testdata/workloads/tpch/queries/tpch-q7.test
M testdata/workloads/tpch/queries/tpch-q8.test
M testdata/workloads/tpch/queries/tpch-q9.test
23 files changed, 63 insertions(+), 19 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/18528/1
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Daniel Becker, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18528
to look at the new patch set (#2).
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
CollectionItemsRead in the runtime profile counts the total number of
nested collection items read by the scan node. Only created for scans
that support nested types, e.g. Parquet or ORC.
Each scanner thread maintains its local counter and merges it into
HdfsScanNode counter for each row batch. However, the local counter in
orc-scanner is uninitialized, leading to weird values. This patch simply
initializes it to 0 and adds test coverage.
Tests:
Add profile verification for this counter on some existing query tests.
Note that there are some implementation difference between Parquet and
ORC scanners (e.g. in predicate pushdown). So we will see different
counter results in some query. I just pick some queries that have
consistent counters.
Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
---
M be/src/exec/hdfs-orc-scanner.h
M testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-array-materialization.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test
M testdata/workloads/tpch/queries/tpch-q1.test
M testdata/workloads/tpch/queries/tpch-q10.test
M testdata/workloads/tpch/queries/tpch-q11.test
M testdata/workloads/tpch/queries/tpch-q12.test
M testdata/workloads/tpch/queries/tpch-q13.test
M testdata/workloads/tpch/queries/tpch-q14.test
M testdata/workloads/tpch/queries/tpch-q15.test
M testdata/workloads/tpch/queries/tpch-q16.test
M testdata/workloads/tpch/queries/tpch-q17.test
M testdata/workloads/tpch/queries/tpch-q18.test
M testdata/workloads/tpch/queries/tpch-q19.test
M testdata/workloads/tpch/queries/tpch-q2.test
M testdata/workloads/tpch/queries/tpch-q20.test
M testdata/workloads/tpch/queries/tpch-q21.test
M testdata/workloads/tpch/queries/tpch-q22.test
M testdata/workloads/tpch/queries/tpch-q3.test
M testdata/workloads/tpch/queries/tpch-q4.test
M testdata/workloads/tpch/queries/tpch-q5.test
M testdata/workloads/tpch/queries/tpch-q6.test
M testdata/workloads/tpch/queries/tpch-q7.test
M testdata/workloads/tpch/queries/tpch-q8.test
M testdata/workloads/tpch/queries/tpch-q9.test
25 files changed, 73 insertions(+), 19 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/18528/2
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18528 )
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18528/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18528/1//COMMIT_MSG@19
PS1, Line 19: There are no nested tables so
: the CollectionItemsRead should all be 0.
This is not clear to me - we don't have any table or tests that would change this to non 0?
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 May 2022 09:21:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18528 )
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10589/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 18 May 2022 04:06:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18528 )
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8116/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 18 May 2022 19:35:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18528 )
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18528/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18528/1//COMMIT_MSG@16
PS1, Line 16: add
> Nit: adds.
Done
http://gerrit.cloudera.org:8080/#/c/18528/1//COMMIT_MSG@19
PS1, Line 19: There are no nested tables so
: the CollectionItemsRead should all be 0.
> This is not clear to me - we don't have any table or tests that would chang
Sorry, I mean the modified tests in this patch set don't query nested tables so the counter should be 0. I should have added tests for non zero cases. Done in the next patch set.
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 18 May 2022 03:47:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18528 )
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10576/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 14 May 2022 07:25:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18528 )
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 18 May 2022 23:59:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18528 )
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
CollectionItemsRead in the runtime profile counts the total number of
nested collection items read by the scan node. Only created for scans
that support nested types, e.g. Parquet or ORC.
Each scanner thread maintains its local counter and merges it into
HdfsScanNode counter for each row batch. However, the local counter in
orc-scanner is uninitialized, leading to weird values. This patch simply
initializes it to 0 and adds test coverage.
Tests:
Add profile verification for this counter on some existing query tests.
Note that there are some implementation difference between Parquet and
ORC scanners (e.g. in predicate pushdown). So we will see different
counter results in some query. I just pick some queries that have
consistent counters.
Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Reviewed-on: http://gerrit.cloudera.org:8080/18528
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/hdfs-orc-scanner.h
M testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-array-materialization.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test
M testdata/workloads/tpch/queries/tpch-q1.test
M testdata/workloads/tpch/queries/tpch-q10.test
M testdata/workloads/tpch/queries/tpch-q11.test
M testdata/workloads/tpch/queries/tpch-q12.test
M testdata/workloads/tpch/queries/tpch-q13.test
M testdata/workloads/tpch/queries/tpch-q14.test
M testdata/workloads/tpch/queries/tpch-q15.test
M testdata/workloads/tpch/queries/tpch-q16.test
M testdata/workloads/tpch/queries/tpch-q17.test
M testdata/workloads/tpch/queries/tpch-q18.test
M testdata/workloads/tpch/queries/tpch-q19.test
M testdata/workloads/tpch/queries/tpch-q2.test
M testdata/workloads/tpch/queries/tpch-q20.test
M testdata/workloads/tpch/queries/tpch-q21.test
M testdata/workloads/tpch/queries/tpch-q22.test
M testdata/workloads/tpch/queries/tpch-q3.test
M testdata/workloads/tpch/queries/tpch-q4.test
M testdata/workloads/tpch/queries/tpch-q5.test
M testdata/workloads/tpch/queries/tpch-q6.test
M testdata/workloads/tpch/queries/tpch-q7.test
M testdata/workloads/tpch/queries/tpch-q8.test
M testdata/workloads/tpch/queries/tpch-q9.test
25 files changed, 73 insertions(+), 19 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18528 )
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 18 May 2022 19:35:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18528 )
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
Patch Set 2: Code-Review+2
Thanks for the new tests!
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 18 May 2022 13:42:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/18528 )
Change subject: IMPALA-11208: Fix uninitialized counter of CollectionItemsRead in orc-scanner
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18528/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18528/1//COMMIT_MSG@16
PS1, Line 16: add
Nit: adds.
--
To view, visit http://gerrit.cloudera.org:8080/18528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7783d1460ac9b98e94d3a31028b43f5a9884f99
Gerrit-Change-Number: 18528
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 May 2022 10:11:24 +0000
Gerrit-HasComments: Yes