You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2022/02/22 18:31:49 UTC
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18265
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Impala crashes on a Parquet file that contains the partition columns.
Data files usually don't contain the partition columns, so Impala don't
expect to find such columns in the data files. Unfortunately min/max
filtering generates a SEGFAULT when the partition column is present in
the data files.
It happens when FindSkipRangesForPagesWithMinMaxFilters() tries to
retrieve the Parquet schema element for a given slot descriptor. When
the slot descriptor refers to a partition column, we usually don't find
a schema element so we don't try to skip pages.
But when the partition column is present in the data file, the code
tries to calculate the filtered pages for the column. It uses the column
reader object corresponding to the column, but this is NULL for
partition columns, hence we get a SEGFAULT.
The code shouldn't do anything at the page-level for partition columns,
as the data in such columns are the same for the whole file and it is
already filtered at a higher level.
Testing:
* added e2e test
Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M testdata/data/README
A testdata/data/partition_col_in_parquet.parquet
M tests/query_test/test_runtime_filters.py
4 files changed, 35 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/18265/1
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18265/2/tests/query_test/test_runtime_filters.py
File tests/query_test/test_runtime_filters.py:
http://gerrit.cloudera.org:8080/#/c/18265/2/tests/query_test/test_runtime_filters.py@328
PS2, Line 328: =
flake8: E225 missing whitespace around operator
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 15:01:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18265/1/tests/query_test/test_runtime_filters.py
File tests/query_test/test_runtime_filters.py:
http://gerrit.cloudera.org:8080/#/c/18265/1/tests/query_test/test_runtime_filters.py@328
PS1, Line 328: =
flake8: E225 missing whitespace around operator
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Feb 2022 18:34:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 21:48:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7872/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 21:48:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7870/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 15:02:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10214/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 15:24:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Qifan Chen, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18265
to look at the new patch set (#3).
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Impala crashes on a Parquet file that contains the partition columns.
Data files usually don't contain the partition columns, so Impala don't
expect to find such columns in the data files. Unfortunately min/max
filtering generates a SEGFAULT when the partition column is present in
the data files.
It happens when FindSkipRangesForPagesWithMinMaxFilters() tries to
retrieve the Parquet schema element for a given slot descriptor. When
the slot descriptor refers to a partition column, we usually don't find
a schema element so we don't try to skip pages.
But when the partition column is present in the data file, the code
tries to calculate the filtered pages for the column. It uses the column
reader object corresponding to the column, but this is NULL for
partition columns, hence we get a SEGFAULT.
The code shouldn't do anything at the page-level for partition columns,
as the data in such columns are the same for the whole file and it is
already filtered at a higher level.
Testing:
* added e2e test
Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M testdata/data/README
A testdata/data/partition_col_in_parquet.parquet
M tests/query_test/test_runtime_filters.py
4 files changed, 35 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/18265/3
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 3: Code-Review+2
Carry +2
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 15:01:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10215/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 15:29:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 1:
Build Successful
https://ec2-35-162-169-52.us-west-2.compute.amazonaws.com/job/gerrit-code-review-checks/10197/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Feb 2022 18:52:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7870/
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 21:41:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Qifan Chen, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18265
to look at the new patch set (#2).
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Impala crashes on a Parquet file that contains the partition columns.
Data files usually don't contain the partition columns, so Impala don't
expect to find such columns in the data files. Unfortunately min/max
filtering generates a SEGFAULT when the partition column is present in
the data files.
It happens when FindSkipRangesForPagesWithMinMaxFilters() tries to
retrieve the Parquet schema element for a given slot descriptor. When
the slot descriptor refers to a partition column, we usually don't find
a schema element so we don't try to skip pages.
But when the partition column is present in the data file, the code
tries to calculate the filtered pages for the column. It uses the column
reader object corresponding to the column, but this is NULL for
partition columns, hence we get a SEGFAULT.
The code shouldn't do anything at the page-level for partition columns,
as the data in such columns are the same for the whole file and it is
already filtered at a higher level.
Testing:
* added e2e test
Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M testdata/data/README
A testdata/data/partition_col_in_parquet.parquet
M tests/query_test/test_runtime_filters.py
4 files changed, 35 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/18265/2
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10201/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Feb 2022 18:55:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Feb 2022 04:24:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Impala crashes on a Parquet file that contains the partition columns.
Data files usually don't contain the partition columns, so Impala don't
expect to find such columns in the data files. Unfortunately min/max
filtering generates a SEGFAULT when the partition column is present in
the data files.
It happens when FindSkipRangesForPagesWithMinMaxFilters() tries to
retrieve the Parquet schema element for a given slot descriptor. When
the slot descriptor refers to a partition column, we usually don't find
a schema element so we don't try to skip pages.
But when the partition column is present in the data file, the code
tries to calculate the filtered pages for the column. It uses the column
reader object corresponding to the column, but this is NULL for
partition columns, hence we get a SEGFAULT.
The code shouldn't do anything at the page-level for partition columns,
as the data in such columns are the same for the whole file and it is
already filtered at a higher level.
Testing:
* added e2e test
Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Reviewed-on: http://gerrit.cloudera.org:8080/18265
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M testdata/data/README
A testdata/data/partition_col_in_parquet.parquet
M tests/query_test/test_runtime_filters.py
4 files changed, 35 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 1:
Thanks for the review!
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 14:57:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7869/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 14:57:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 )
Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
......................................................................
Patch Set 1: Code-Review+2
Thanks a lot Zoltan for fixing this bug!
--
To view, visit http://gerrit.cloudera.org:8080/18265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Gerrit-Change-Number: 18265
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 14:50:21 +0000
Gerrit-HasComments: No