You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2022/07/04 15:24:38 UTC

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18700


Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................

IMPALA-11414: Off-by-one error in Parquet late materialization

With PARQUET_LATE_MATERIALIZATION we can set the number of minimum
consecutive rows that if filtered out, we avoid materialization of rows
in other columns in parquet.

E.g. if PARQUET_LATE_MATERIALIZATION is 10, and in a filtered column we
find at least 10 consecutive rows that don't pass the predicates we
avoid materializing the corresponding rows in the other columns.

But due to an off-by-one error we actually only needed
(PARQUET_LATE_MATERIALIZATION - 1) consecutive elements. This means if
we set PARQUET_LATE_MATERIALIZATION to one, then we need zero
consecutive filtered out elements which leads to a crash/DCHECK. The bug
is in the GetMicroBatches() algorithm when we produce the micro batches
based on the selected rows.

Setting PARQUET_LATE_MATERIALIZATION to 0 doesn't make sense so it
shouldn't be allowed.

Testing
 * e2e test with PARQUET_LATE_MATERIALIZATION=1
 * e2e test for checking SET PARQUET_LATE_MATERIALIZATION=N

Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
---
M be/src/exec/scratch-tuple-batch.h
M be/src/service/query-options.cc
A testdata/workloads/functional-query/queries/QueryTest/parquet-late-materialization-unique-db.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/query_test/test_parquet_late_materialization.py
5 files changed, 45 insertions(+), 7 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/18700/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18700 )

Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10930/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 06 Jul 2022 11:11:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18700

to look at the new patch set (#2).

Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................

IMPALA-11414: Off-by-one error in Parquet late materialization

With PARQUET_LATE_MATERIALIZATION we can set the number of minimum
consecutive rows that if filtered out, we avoid materialization of rows
in other columns in parquet.

E.g. if PARQUET_LATE_MATERIALIZATION is 10, and in a filtered column we
find at least 10 consecutive rows that don't pass the predicates we
avoid materializing the corresponding rows in the other columns.

But due to an off-by-one error we actually only needed
(PARQUET_LATE_MATERIALIZATION - 1) consecutive elements. This means if
we set PARQUET_LATE_MATERIALIZATION to one, then we need zero
consecutive filtered out elements which leads to a crash/DCHECK. The bug
is in the GetMicroBatches() algorithm when we produce the micro batches
based on the selected rows.

Setting PARQUET_LATE_MATERIALIZATION to 0 doesn't make sense so it
shouldn't be allowed.

Testing
 * e2e test with PARQUET_LATE_MATERIALIZATION=1
 * e2e test for checking SET PARQUET_LATE_MATERIALIZATION=N

Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
---
M be/src/exec/scratch-tuple-batch-test.cc
M be/src/exec/scratch-tuple-batch.h
M be/src/service/query-options.cc
A testdata/workloads/functional-query/queries/QueryTest/parquet-late-materialization-unique-db.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/query_test/test_parquet_late_materialization.py
6 files changed, 46 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/18700/2
-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18700 )

Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................

IMPALA-11414: Off-by-one error in Parquet late materialization

With PARQUET_LATE_MATERIALIZATION we can set the number of minimum
consecutive rows that if filtered out, we avoid materialization of rows
in other columns in parquet.

E.g. if PARQUET_LATE_MATERIALIZATION is 10, and in a filtered column we
find at least 10 consecutive rows that don't pass the predicates we
avoid materializing the corresponding rows in the other columns.

But due to an off-by-one error we actually only needed
(PARQUET_LATE_MATERIALIZATION - 1) consecutive elements. This means if
we set PARQUET_LATE_MATERIALIZATION to one, then we need zero
consecutive filtered out elements which leads to a crash/DCHECK. The bug
is in the GetMicroBatches() algorithm when we produce the micro batches
based on the selected rows.

Setting PARQUET_LATE_MATERIALIZATION to 0 doesn't make sense so it
shouldn't be allowed.

Testing
 * e2e test with PARQUET_LATE_MATERIALIZATION=1
 * e2e test for checking SET PARQUET_LATE_MATERIALIZATION=N

Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Reviewed-on: http://gerrit.cloudera.org:8080/18700
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/scratch-tuple-batch-test.cc
M be/src/exec/scratch-tuple-batch.h
M be/src/service/query-options.cc
A testdata/workloads/functional-query/queries/QueryTest/parquet-late-materialization-unique-db.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/query_test/test_parquet_late_materialization.py
6 files changed, 46 insertions(+), 8 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18700 )

Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Jul 2022 14:41:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18700 )

Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Jul 2022 19:25:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/18700 )

Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................


Patch Set 1: Code-Review+1

Looks good, I'll give a chance for others to review it too as I'm not familiar with these places in the code.


-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 05 Jul 2022 08:41:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18700 )

Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................


Patch Set 2:

Thanks for the review!


-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Jul 2022 14:41:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18700 )

Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8303/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Jul 2022 14:41:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18700 )

Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................


Patch Set 2: Code-Review+2

Pretty nice fix! Carrying Daniel's +1.


-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Jul 2022 09:42:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11414: Off-by-one error in Parquet late materialization

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18700 )

Change subject: IMPALA-11414: Off-by-one error in Parquet late materialization
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10919/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18700
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I38f95ad48c4ac8c1e06651565ab5c496283b29fa
Gerrit-Change-Number: 18700
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Jul 2022 15:45:38 +0000
Gerrit-HasComments: No