You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Xuebin Su (Code Review)" <ge...@cloudera.org> on 2025/06/03 03:43:46 UTC

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Xuebin Su has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22974


Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................

IMPALA-14110: Avoid decoding values for counting columns

For a counting column, its slot descriptor is null and its data decoder
is not initialized. Therefore, trying to decode the values when skipping
them will lead to check failure.

This patch fixes the issue by returning early when trying to skip values
if the current column is a counting column to avoid trying to decode any
value.

Testing:
- Passed the previously failed Parquet tests in exhaustive mode.

Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-column-readers.cc
M testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test
3 files changed, 8 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/22974/1
-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 1
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 5
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 22:20:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19371/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 2
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 07:50:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 1: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22974/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/22974/1//COMMIT_MSG@18
PS1, Line 18: Passed the previously failed Parquet tests in exhaustive mode.
            : 
nit: Passed TestZippingUnnest in exhaustive mode.



-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 1
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 03:54:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 4: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22974/3/testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test
File testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test:

http://gerrit.cloudera.org:8080/#/c/22974/3/testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test@752
PS3, Line 752: ---- QUERY
             : # Test if page filtering works for counting columns when late materialization
             : # is disabled.
             : set PARQUET_LATE_MATERIALIZATION_THRESHOLD = -1;
             : select d_38 from nested_decimals n, n.arr where d_38 > 7;
> Thanks! Added.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 4
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 01:54:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19369/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 1
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 04:10:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19381/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 4
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 02:18:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 4: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 4
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 22:19:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11971/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 5
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 22:20:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Xuebin Su (Code Review)" <ge...@cloudera.org>.
Xuebin Su has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 4:

(1 comment)

> Patch Set 3: Code-Review+1
> 
> (1 comment)

Thanks for reviewing!

http://gerrit.cloudera.org:8080/#/c/22974/3/testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test
File testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test:

http://gerrit.cloudera.org:8080/#/c/22974/3/testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test@752
PS3, Line 752: ---- QUERY
             : # Test if page filtering works for counting columns when late materialization
             : # is disabled.
             : set PARQUET_LATE_MATERIALIZATION_THRESHOLD = -1;
             : select d_38 from nested_decimals n, n.arr where d_38 > 7;
> Please add this test for completeness.
Thanks! Added.



-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 4
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 01:52:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Xuebin Su (Code Review)" <ge...@cloudera.org>.
Xuebin Su has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................

IMPALA-14110: Avoid decoding values for counting columns

For a counting column, its slot descriptor is null and its data decoder
is not initialized. Therefore, trying to decode the values when skipping
them will lead to check failure.

This patch fixes the issue by returning early when trying to skip values
if the current column is a counting column to avoid trying to decode any
value.

Testing:
- Passed TestZippingUnnest in exhaustive mode.
- Added a test case to make sure that page filtering works for counting
  columns.

Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-column-readers.cc
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test
M testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test
4 files changed, 19 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/22974/3
-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 3
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19375/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 3
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 12:06:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 5
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jun 2025 04:17:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................

IMPALA-14110: Avoid decoding values for counting columns

For a counting column, its slot descriptor is null and its data decoder
is not initialized. Therefore, trying to decode the values when skipping
them will lead to check failure.

This patch fixes the issue by returning early when trying to skip values
if the current column is a counting column to avoid trying to decode any
value.

Testing:
- Passed TestZippingUnnest in exhaustive mode.
- Added test cases to make sure that page filtering works for counting
  columns.

Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Reviewed-on: http://gerrit.cloudera.org:8080/22974
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-column-readers.cc
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test
M testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test
4 files changed, 31 insertions(+), 1 deletion(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 6
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Xuebin Su (Code Review)" <ge...@cloudera.org>.
Xuebin Su has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 3:

(3 comments)

> Patch Set 1: Code-Review+1
> 
> (2 comments)

Thanks for reviewing!

http://gerrit.cloudera.org:8080/#/c/22974/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/22974/1//COMMIT_MSG@18
PS1, Line 18: Passed TestZippingUnnest in exhaustive mode.
            : 
> nit: Passed TestZippingUnnest in exhaustive mode.
Thanks! Changed.


http://gerrit.cloudera.org:8080/#/c/22974/1/testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test
File testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test:

http://gerrit.cloudera.org:8080/#/c/22974/1/testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test@288
PS1, Line 288: # Similar as above but there is a where clause on a non-array field.
> Not necessarily in this commit, but it could be useful to add a test dimens
Sure. Thanks! I think we can do it later.


http://gerrit.cloudera.org:8080/#/c/22974/1/testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test@291
PS1, Line 291: select id from complextypes_arrays t, unnest(t.arr1) where id = 7;
> I am trying to understand what this specific query does - it only returns i
I agree with you about what this query does.

I also agree that this is an old bug that affects page filtering. Added a test case for that.



-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 3
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 11:44:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Xuebin Su (Code Review)" <ge...@cloudera.org>.
Xuebin Su has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................

IMPALA-14110: Avoid decoding values for counting columns

For a counting column, its slot descriptor is null and its data decoder
is not initialized. Therefore, trying to decode the values when skipping
them will lead to check failure.

This patch fixes the issue by returning early when trying to skip values
if the current column is a counting column to avoid trying to decode any
value.

Testing:
- Passed TestZippingUnnest in exhaustive mode.
- Added test cases to make sure that page filtering works for counting
  columns.

Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-column-readers.cc
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test
M testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test
4 files changed, 31 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/22974/4
-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 4
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 3: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22974/3/testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test
File testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test:

http://gerrit.cloudera.org:8080/#/c/22974/3/testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test@752
PS3, Line 752: ---- QUERY
             : # Test if page filtering works for counting columns when late materialization
             : # is disabled.
             : set PARQUET_LATE_MATERIALIZATION_THRESHOLD = -1;
             : select d_38 from nested_decimals n, n.arr where d_38 > 7;
Please add this test for completeness.

---- QUERY
# Test if page filtering works for counting columns when late materialization
# is enabled (the default).
set PARQUET_LATE_MATERIALIZATION_THRESHOLD = "";
select d_38 from nested_decimals n, n.arr where d_38 > 7;



-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 3
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 14:43:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................


Patch Set 1: Code-Review+1

(2 comments)

http://gerrit.cloudera.org:8080/#/c/22974/1/testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test
File testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test:

http://gerrit.cloudera.org:8080/#/c/22974/1/testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test@288
PS1, Line 288: # Similar as above but there is a where clause on a non-array field.
Not necessarily in this commit, but it could be useful to add a test dimension for late materialization in https://github.com/apache/impala/blob/master/tests/query_test/test_nested_types.py


http://gerrit.cloudera.org:8080/#/c/22974/1/testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test@291
PS1, Line 291: select id from complextypes_arrays t, unnest(t.arr1) where id = 7;
I am trying to understand what this specific query does - it only returns id, but repeats it for each row depending on the length of arr1, which means that arr1 we don't need the actual value, only some placeholder for the value, right?

I am wondering if this was also reproducible before IMPALA-3841, as skipping already existed before lazy materialization to implement page filtering. My guess is that this is an old bug, just there wasn't any test for page filtering + complex types.



-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 1
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 06:44:55 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14110: Avoid decoding values for counting columns

Posted by "Xuebin Su (Code Review)" <ge...@cloudera.org>.
Xuebin Su has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/22974 )

Change subject: IMPALA-14110: Avoid decoding values for counting columns
......................................................................

IMPALA-14110: Avoid decoding values for counting columns

For a counting column, its slot descriptor is null and its data decoder
is not initialized. Therefore, trying to decode the values when skipping
them will lead to check failure.

This patch fixes the issue by returning early when trying to skip values
if the current column is a counting column to avoid trying to decode any
value.

Testing:
- Passed TestZippingUnnest in exhaustive mode.

Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-column-readers.cc
M testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test
3 files changed, 8 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/22974/2
-- 
To view, visit http://gerrit.cloudera.org:8080/22974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia707335c50cc0653097f375aae3f10609e0eb091
Gerrit-Change-Number: 22974
Gerrit-PatchSet: 2
Gerrit-Owner: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>