You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Qifan Chen (Code Review)" <ge...@cloudera.org> on 2021/06/25 00:25:56 UTC
[Impala-ASF-CR] [WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17635
Change subject: [WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
[WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
This patch enables min/max filtering on Z-order sorted by columns by
default.
Since the column stats for a row group or a page is computed from the
column valuess stored in the row group or the page, the current
infrastructure for min/max filtering works for the Z-order out of box.
The fact that these column valuess are ordered by Z-order is
orthogonal to the work of min/max filtering.
Testing [TBD]
core-test
Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
---
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
1 file changed, 12 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/17635/1
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 1
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: [WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9004/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 1
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Jun 2021 00:47:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 8: Code-Review+2
(4 comments)
Found few nits, but LGTM!
http://gerrit.cloudera.org:8080/#/c/17635/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17635/8//COMMIT_MSG@13
PS8, Line 13: valuess
values
http://gerrit.cloudera.org:8080/#/c/17635/8//COMMIT_MSG@15
PS8, Line 15: valuess
values
http://gerrit.cloudera.org:8080/#/c/17635/8/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java:
http://gerrit.cloudera.org:8080/#/c/17635/8/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@304
PS8, Line 304: -1
Do we need to set the limit parameter for split()?
http://gerrit.cloudera.org:8080/#/c/17635/8/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/17635/8/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@729
PS8, Line 729: allowRuntimeFilter
nit: Since this mmethod is only about Min/Max filters, and not bloom filters, probably we should rename it to allowMinMaxFilter()
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 8
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Jul 2021 12:17:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
This patch enables min/max filtering on any Z-order sort-by columns
by default.
Since the column stats for a row group or a page is computed from the
column valuess stored in the row group or the page, the current
infrastructure for min/max filtering works for the Z-order out of box.
The fact that these column valuess are ordered by Z-order is
orthogonal to the work of min/max filtering.
By default, the new feature is enabled. Set the existing control knob
minmax_filter_sorted_columns to false to turn it off.
Testing
1. Added new z-order related sort column tests in
overlap_min_max_filters_on_sorted_columns.test;
2. Ran core-test.
Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
---
M be/src/service/query-options.cc
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
4 files changed, 145 insertions(+), 27 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/17635/8
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 8
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 10: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 10
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Jul 2021 19:07:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
This patch enables min/max filtering on Z-order sorted by columns by
default.
Since the column stats for a row group or a page is computed from the
column valuess stored in the row group or the page, the current
infrastructure for min/max filtering works for the Z-order out of box.
The fact that these column valuess are ordered by Z-order is
orthogonal to the work of min/max filtering.
Testing
1. New z-order related sort column tests in
overlap_min_max_filters_on_sorted_columns.test;
2. core-test [TBD]
Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
---
M be/src/service/query-options.cc
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
4 files changed, 145 insertions(+), 27 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/17635/7
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 7
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
This patch enables min/max filtering on Z-order sorted by columns by
default.
Since the column stats for a row group or a page is computed from the
column valuess stored in the row group or the page, the current
infrastructure for min/max filtering works for the Z-order out of box.
The fact that these column valuess are ordered by Z-order is
orthogonal to the work of min/max filtering.
Testing [TBD]
1. New z-order related sort column tests in
overlap_min_max_filters_on_sorted_columns.test;
2. core-test
Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
---
M be/src/service/query-options.cc
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
4 files changed, 119 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/17635/4
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 4
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 6:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/9064/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 6
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 09 Jul 2021 18:10:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
This patch enables min/max filtering on any Z-order sort-by columns
by default.
Since the column stats for a row group or a page is computed from the
column values stored in the row group or the page, the current
infrastructure for min/max filtering works for the Z-order out of box.
The fact that these column values are ordered by Z-order is
orthogonal to the work of min/max filtering.
By default, the new feature is enabled. Set the existing control knob
minmax_filter_sorted_columns to false to turn it off.
Testing
1. Added new z-order related sort column tests in
overlap_min_max_filters_on_sorted_columns.test;
2. Ran core-test.
Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
---
M be/src/service/query-options.cc
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
4 files changed, 146 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/17635/9
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 9
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
This patch enables min/max filtering on Z-order sorted by columns by
default.
Since the column stats for a row group or a page is computed from the
column valuess stored in the row group or the page, the current
infrastructure for min/max filtering works for the Z-order out of box.
The fact that these column valuess are ordered by Z-order is
orthogonal to the work of min/max filtering.
Testing
1. New z-order related sort column tests in
overlap_min_max_filters_on_sorted_columns.test;
2. core-test [TBD]
Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
---
M be/src/service/query-options.cc
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
4 files changed, 172 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/17635/5
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] [WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: [WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9005/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Jun 2021 00:52:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9066/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 7
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 09 Jul 2021 20:29:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/17635 )
Change subject: [WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
[WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
This patch enables min/max filtering on Z-order sorted by columns by
default.
Since the column stats for a row group or a page is computed from the
column valuess stored in the row group or the page, the current
infrastructure for min/max filtering works for the Z-order out of box.
The fact that these column valuess are ordered by Z-order is
orthogonal to the work of min/max filtering.
Testing [TBD]
core-test
Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
---
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
1 file changed, 12 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/17635/2
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9069/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 8
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 11 Jul 2021 12:52:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7303/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 10
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Jul 2021 12:57:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 5:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/9063/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 09 Jul 2021 17:43:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 10: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 10
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Jul 2021 12:57:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9062/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 4
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 09 Jul 2021 13:50:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9033/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 05 Jul 2021 16:17:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
This patch enables min/max filtering on any Z-order sort-by columns
by default.
Since the column stats for a row group or a page is computed from the
column values stored in the row group or the page, the current
infrastructure for min/max filtering works for the Z-order out of box.
The fact that these column values are ordered by Z-order is
orthogonal to the work of min/max filtering.
By default, the new feature is enabled. Set the existing control knob
minmax_filter_sorted_columns to false to turn it off.
Testing
1. Added new z-order related sort column tests in
overlap_min_max_filters_on_sorted_columns.test;
2. Ran core-test.
Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Reviewed-on: http://gerrit.cloudera.org:8080/17635
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/query-options.cc
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
4 files changed, 146 insertions(+), 28 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 11
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: [WIP] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 2:
(2 comments)
Good to see the enhancement :)
http://gerrit.cloudera.org:8080/#/c/17635/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/17635/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@752
PS2, Line 752: table != null
nit: this can be removed since null can't satify "table instanceof FeFsTable"
http://gerrit.cloudera.org:8080/#/c/17635/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@762
PS2, Line 762: // Must be Z-order. Allow it.
: Preconditions.checkState(sortOrder == TSortingOrder.ZORDER);
Should we check whether the column is in the "sort.columns"? Z-order doesn't sort on all columns.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 25 Jun 2021 08:28:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9086/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 9
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Jul 2021 16:31:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
This patch enables min/max filtering on Z-order sorted by columns by
default.
Since the column stats for a row group or a page is computed from the
column valuess stored in the row group or the page, the current
infrastructure for min/max filtering works for the Z-order out of box.
The fact that these column valuess are ordered by Z-order is
orthogonal to the work of min/max filtering.
Testing [TBD]
1. New z-order related sort column tests in
overlap_min_max_filters_on_sorted_columns.test;
2. core-test
Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
---
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
3 files changed, 116 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/17635/3
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 9: Code-Review+2
Thanks for applying the changes!
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 9
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 14 Jul 2021 13:29:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
This patch enables min/max filtering on Z-order sorted by columns by
default.
Since the column stats for a row group or a page is computed from the
column valuess stored in the row group or the page, the current
infrastructure for min/max filtering works for the Z-order out of box.
The fact that these column valuess are ordered by Z-order is
orthogonal to the work of min/max filtering.
Testing
1. New z-order related sort column tests in
overlap_min_max_filters_on_sorted_columns.test;
2. core-test [TBD]
Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
---
M be/src/service/query-options.cc
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
4 files changed, 150 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/17635/6
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 6
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17635 )
Change subject: IMPALA-10763: Min/max filters should be enabled on Z-order sorted columns
......................................................................
Patch Set 8: Code-Review+2
(1 comment)
LGTM
http://gerrit.cloudera.org:8080/#/c/17635/8/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
File testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test:
http://gerrit.cloudera.org:8080/#/c/17635/8/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test@284
PS8, Line 284: PRIMARY KEY (ss_item_sk, ss_ticket_number)
nit: PRIMARY KEY seems useless here. I think it's only used by Kudu tables.
--
To view, visit http://gerrit.cloudera.org:8080/17635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a528ffbd0e333721ef38b4be7d4ddcdbf188adf
Gerrit-Change-Number: 17635
Gerrit-PatchSet: 8
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 11 Jul 2021 13:40:11 +0000
Gerrit-HasComments: Yes