You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "rtrivedi12 (via GitHub)" <gi...@apache.org> on 2026/07/13 16:24:40 UTC
[PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
rtrivedi12 opened a new pull request, #6601:
URL: https://github.com/apache/hive/pull/6601
…ng for filters over windowing Projects
### What changes were proposed in this pull request?
RedundancyChecker.visit(...) in HiveFilterProjectTransposeRule now skips (returns) when it reaches a Project that contains window functions (RexOver), instead of calling pushPastProjectUnlessBloat on it. This mirrors the guard that already exists in onMatch(...) and the visitor's existing "unsupported node → return" behaviour
### Why are the changes needed?
isRedundantIsNotNull(...) (added in HIVE-25275) walks down the plan and pushes the predicate past each Project. For a Project holding RexOver expressions, Calcite's pushPastProjectUnlessBloat rewrites the RexInputRefs inside the window PARTITION BY / ORDER BY specs; those refs index into the input row type and can exceed the projection list size, throwing ArrayIndexOutOfBoundsException during CBO planning
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dqfile=cbo_filter_pushdown_windowing_notnull.q
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "rtrivedi12 (via GitHub)" <gi...@apache.org>.
rtrivedi12 commented on code in PR #6601:
URL: https://github.com/apache/hive/pull/6601#discussion_r3581224917
##########
ql/src/test/queries/clientpositive/cbo_filter_pushdown_windowing_notnull.q:
##########
@@ -0,0 +1,88 @@
+-- HIVE-29729: an IS NOT NULL predicate above nested windowing (OVER) Projects must
+-- not trip the RedundancyChecker in HiveFilterProjectTransposeRule.
+set hive.cbo.fallback.strategy=NEVER;
+
+create table cl_wl_daily_table (
+ user_id string,
+ lender string,
+ attributes string,
+ loan_type string,
+ whitelisting_status string
+) stored as orc;
+
+explain
+select
+ user_id,
+ cg_tg_cohort
+from (
+ select
+ user_id,
+ lender,
+ maximum_principal,
+ loan_type,
+ case
+ when loan_type = 'FRESH' and rnk <= (333333 / user_count) * 0.1 then 'CG'
+ when loan_type = 'FRESH' and rnk <= (333333 / user_count) then 'TG'
+ when loan_type = 'REPEAT' and rnk <= (500 / user_count)
+ and substring(user_id, -2) between '60' and '69' then 'CG'
+ when loan_type = 'REPEAT' and rnk <= (500 / user_count)
+ and substring(user_id, -2) between '00' and '45' then 'TG'
+ when loan_type = 'REPEAT' and rnk <= (500 / user_count) then 'TG-TELE'
+ end as cg_tg_cohort
+ from (
+ select
+ user_id,
+ lender,
+ maximum_principal,
+ loan_type,
+ percent_rank() over(partition by loan_type order by rnk) as rnk,
+ count(user_id) over(partition by loan_type) as user_count
+ from (
+ select
+ user_id,
+ lender,
+ maximum_principal,
+ loan_type,
+ row_number() over(partition by loan_type order by rand()) as rnk
+ from (
+ select
+ wl.user_id as user_id,
+ max(lender) as lender,
+ max(maximum_principal) as maximum_principal,
+ max(loan_type) as loan_type
+ from (
+ select
+ user_id,
+ lender,
+ case
+ when repeat_wl = 0 and platform_repeat_wl = 0 then 'FRESH'
+ else 'REPEAT'
+ end as loan_type,
+ cast(maximum_principal / 100 as int) as maximum_principal
+ from (
+ select
+ user_id,
+ concat_ws('#', sort_array(collect_set(lender))) as lender,
+ max(
+ case
+ when coalesce(get_json_object(attributes, '$.maximum_principal'), '_') = '_' then 0
+ else get_json_object(attributes, '$.maximum_principal')
+ end
+ ) as maximum_principal,
+ max(case when loan_type = 'REPEAT' then 1 else 0 end) as repeat_wl,
+ max(case when loan_type = 'PLATFORM_REPEAT' then 1 else 0 end) as platform_repeat_wl
+ from cl_wl_daily_table
+ where whitelisting_status = 'Currently WL'
+ group by user_id
+ ) base
+ where maximum_principal > 0
+ and maximum_principal is not null
+ and maximum_principal <> '_'
+ ) wl
+ group by
+ wl.user_id
+ ) grouped_users
+ ) ranked_once
+ ) ranked_twice
+) cohorted
+where cg_tg_cohort is not null;
Review Comment:
Thank you so much for the review @thomasrebele and short repro!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6601:
URL: https://github.com/apache/hive/pull/6601#issuecomment-4972889657
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6601) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6601&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6601&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6601&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6601&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6601&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6601)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "rtrivedi12 (via GitHub)" <gi...@apache.org>.
rtrivedi12 commented on code in PR #6601:
URL: https://github.com/apache/hive/pull/6601#discussion_r3588912830
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java:
##########
@@ -338,11 +338,14 @@ public void visit(RelNode node, int ordinal, RelNode parent) {
if (node instanceof Filter) {
check((Filter) node);
} else if (node instanceof Project) {
+ Project project = (Project) node;
RexNode condition = HiveRelOptUtil.pushPastProjectUnlessBloat(
- filterCondition, (Project) node, bloat);
- if (condition != null) {
- filterCondition = condition;
+ filterCondition, project, bloat);
+ if (condition == null) {
+ // the condition could not be pushed, so bail out
+ return;
}
+ filterCondition = condition;
Review Comment:
Thanks for the feedback @soumyakanti3578 ! I will update PR with these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6601:
URL: https://github.com/apache/hive/pull/6601#issuecomment-4983517581
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6601) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6601&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6601&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6601&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6601&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6601&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6601)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "soumyakanti3578 (via GitHub)" <gi...@apache.org>.
soumyakanti3578 commented on code in PR #6601:
URL: https://github.com/apache/hive/pull/6601#discussion_r3584419586
##########
ql/src/test/queries/clientpositive/cbo_filter_pushdown_windowing_notnull.q:
##########
@@ -0,0 +1,16 @@
+-- HIVE-29729: an IS NOT NULL predicate above nested windowing (OVER) Projects must
+-- not trip the RedundancyChecker in HiveFilterProjectTransposeRule.
+set hive.cbo.fallback.strategy=NEVER;
Review Comment:
nit: Is this needed? I think `NEVER` is the default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6601:
URL: https://github.com/apache/hive/pull/6601#discussion_r3578309543
##########
ql/src/test/queries/clientpositive/cbo_filter_pushdown_windowing_notnull.q:
##########
@@ -0,0 +1,88 @@
+-- HIVE-29729: an IS NOT NULL predicate above nested windowing (OVER) Projects must
+-- not trip the RedundancyChecker in HiveFilterProjectTransposeRule.
+set hive.cbo.fallback.strategy=NEVER;
+
+create table cl_wl_daily_table (
+ user_id string,
+ lender string,
+ attributes string,
+ loan_type string,
+ whitelisting_status string
+) stored as orc;
+
+explain
+select
+ user_id,
+ cg_tg_cohort
+from (
+ select
+ user_id,
+ lender,
+ maximum_principal,
+ loan_type,
+ case
+ when loan_type = 'FRESH' and rnk <= (333333 / user_count) * 0.1 then 'CG'
+ when loan_type = 'FRESH' and rnk <= (333333 / user_count) then 'TG'
+ when loan_type = 'REPEAT' and rnk <= (500 / user_count)
+ and substring(user_id, -2) between '60' and '69' then 'CG'
+ when loan_type = 'REPEAT' and rnk <= (500 / user_count)
+ and substring(user_id, -2) between '00' and '45' then 'TG'
+ when loan_type = 'REPEAT' and rnk <= (500 / user_count) then 'TG-TELE'
+ end as cg_tg_cohort
+ from (
+ select
+ user_id,
+ lender,
+ maximum_principal,
+ loan_type,
+ percent_rank() over(partition by loan_type order by rnk) as rnk,
+ count(user_id) over(partition by loan_type) as user_count
+ from (
+ select
+ user_id,
+ lender,
+ maximum_principal,
+ loan_type,
+ row_number() over(partition by loan_type order by rand()) as rnk
+ from (
+ select
+ wl.user_id as user_id,
+ max(lender) as lender,
+ max(maximum_principal) as maximum_principal,
+ max(loan_type) as loan_type
+ from (
+ select
+ user_id,
+ lender,
+ case
+ when repeat_wl = 0 and platform_repeat_wl = 0 then 'FRESH'
+ else 'REPEAT'
+ end as loan_type,
+ cast(maximum_principal / 100 as int) as maximum_principal
+ from (
+ select
+ user_id,
+ concat_ws('#', sort_array(collect_set(lender))) as lender,
+ max(
+ case
+ when coalesce(get_json_object(attributes, '$.maximum_principal'), '_') = '_' then 0
+ else get_json_object(attributes, '$.maximum_principal')
+ end
+ ) as maximum_principal,
+ max(case when loan_type = 'REPEAT' then 1 else 0 end) as repeat_wl,
+ max(case when loan_type = 'PLATFORM_REPEAT' then 1 else 0 end) as platform_repeat_wl
+ from cl_wl_daily_table
+ where whitelisting_status = 'Currently WL'
+ group by user_id
+ ) base
+ where maximum_principal > 0
+ and maximum_principal is not null
+ and maximum_principal <> '_'
+ ) wl
+ group by
+ wl.user_id
+ ) grouped_users
+ ) ranked_once
+ ) ranked_twice
+) cohorted
+where cg_tg_cohort is not null;
Review Comment:
Could you please replace the q file test with a minimal one that reproduces the exception? How about the following q file test?
```
-- HIVE-29729: an IS NOT NULL predicate above nested windowing (OVER) Projects must
-- not trip the RedundancyChecker in HiveFilterProjectTransposeRule.
set hive.cbo.fallback.strategy=NEVER;
create table tab1 (id string);
select rnk2
from (
select
row_number() over(order by rnk) as rnk2
from (
select count(*) over() as rnk
from (select count(*) from tab1) t
) t2
) t3
where rnk2 is not null;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6601:
URL: https://github.com/apache/hive/pull/6601#issuecomment-4985063552
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6601) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6601&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6601&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6601&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6601&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6601&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6601)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6601:
URL: https://github.com/apache/hive/pull/6601#discussion_r3580218685
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java:
##########
@@ -338,8 +338,18 @@ public void visit(RelNode node, int ordinal, RelNode parent) {
if (node instanceof Filter) {
check((Filter) node);
} else if (node instanceof Project) {
+ Project project = (Project) node;
+ if (RexOver.containsOver(project.getProjects(), null)) {
+ // RelOptUtil.pushPastProjectUnlessBloat may rewrite RexInputRefs inside RexOver
+ // window specs using the Project expression list. Window partition/order
+ // expressions reference the Project input row type, so their indexes may fall
+ // outside project.getProjects(). This visitor is only a conservative redundancy
+ // check for IS NOT NULL predicates; if it cannot safely inspect through a
+ // windowing Project, stop the check instead of failing query planning.
+ return;
+ }
RexNode condition = HiveRelOptUtil.pushPastProjectUnlessBloat(
- filterCondition, (Project) node, bloat);
+ filterCondition, project, bloat);
if (condition != null) {
filterCondition = condition;
Review Comment:
I did a bit of debugging, and the problem might be around here. pushPastProjectUnlessBloat checks for OVER and returns null. The filterCondition is not updated, so it continues to hold the filter condition of the parent.
For the q test file, the filterCondition corresponding to the parent is `IS NOT NULL(row_number() OVER (PARTITION BY 0 ORDER BY $1 NULLS LAST RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING))`. This will be also applied to the child, which is `HiveProject(_o__c0=[$0])`. So a project with just one field, making the $1 RexInputRef illegal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "soumyakanti3578 (via GitHub)" <gi...@apache.org>.
soumyakanti3578 commented on code in PR #6601:
URL: https://github.com/apache/hive/pull/6601#discussion_r3584416153
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java:
##########
@@ -338,11 +338,14 @@ public void visit(RelNode node, int ordinal, RelNode parent) {
if (node instanceof Filter) {
check((Filter) node);
} else if (node instanceof Project) {
+ Project project = (Project) node;
RexNode condition = HiveRelOptUtil.pushPastProjectUnlessBloat(
- filterCondition, (Project) node, bloat);
- if (condition != null) {
- filterCondition = condition;
+ filterCondition, project, bloat);
+ if (condition == null) {
+ // the condition could not be pushed, so bail out
+ return;
}
+ filterCondition = condition;
Review Comment:
nit: This can be rewritten as
```
node instanceof Project project
```
and also I think `filterCondition` can be directly assigned `HiveRelOptUtil.pushPastProjectUnlessBloat(
filterCondition, project, bloat);` without any side effects?
Neither of these are needed for the PR btw so I will leave it up to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6601:
URL: https://github.com/apache/hive/pull/6601#issuecomment-4961048124
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6601) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6601&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6601&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6601&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6601&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6601&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6601)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "rtrivedi12 (via GitHub)" <gi...@apache.org>.
rtrivedi12 commented on code in PR #6601:
URL: https://github.com/apache/hive/pull/6601#discussion_r3581236338
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java:
##########
@@ -338,8 +338,18 @@ public void visit(RelNode node, int ordinal, RelNode parent) {
if (node instanceof Filter) {
check((Filter) node);
} else if (node instanceof Project) {
+ Project project = (Project) node;
+ if (RexOver.containsOver(project.getProjects(), null)) {
+ // RelOptUtil.pushPastProjectUnlessBloat may rewrite RexInputRefs inside RexOver
+ // window specs using the Project expression list. Window partition/order
+ // expressions reference the Project input row type, so their indexes may fall
+ // outside project.getProjects(). This visitor is only a conservative redundancy
+ // check for IS NOT NULL predicates; if it cannot safely inspect through a
+ // windowing Project, stop the check instead of failing query planning.
+ return;
+ }
RexNode condition = HiveRelOptUtil.pushPastProjectUnlessBloat(
- filterCondition, (Project) node, bloat);
+ filterCondition, project, bloat);
if (condition != null) {
filterCondition = condition;
Review Comment:
I see .. The real defect was the condition == null path: when pushPastProjectUnlessBloat couldn't push the predicate, the old code kept the un-translated condition and still descended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "rtrivedi12 (via GitHub)" <gi...@apache.org>.
rtrivedi12 commented on code in PR #6601:
URL: https://github.com/apache/hive/pull/6601#discussion_r3588920518
##########
ql/src/test/queries/clientpositive/cbo_filter_pushdown_windowing_notnull.q:
##########
@@ -0,0 +1,16 @@
+-- HIVE-29729: an IS NOT NULL predicate above nested windowing (OVER) Projects must
+-- not trip the RedundancyChecker in HiveFilterProjectTransposeRule.
+set hive.cbo.fallback.strategy=NEVER;
Review Comment:
Yes, NEVER is the new default but I kept it for test intent. And also it makes running the qtest in other branches where CONSERVATIVE is default easy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29729: AIBOE in HiveFilterProjectTransposeRule during CBO planni… [hive]
Posted by "soumyakanti3578 (via GitHub)" <gi...@apache.org>.
soumyakanti3578 merged PR #6601:
URL: https://github.com/apache/hive/pull/6601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org