You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "OIiveirra (via GitHub)" <gi...@apache.org> on 2026/10/14 12:08:58 UTC
[PR] [fix](nereids) Preserve date arithmetic evaluation boundaries [doris]
OIiveirra opened a new pull request, #69073:
URL: https://github.com/apache/doris/pull/69073
### What problem does this PR solve?
Issue Number: DORIS-29604 (Jira)
Problem Summary:
An outer predicate on a date arithmetic alias could be expanded and pushed ahead of an inner filter. This evaluated `WEEKS_ADD('9999-12-31 23:59:59', 5)` on rows that the inner `WHERE` should discard, causing E218 instead of the expected empty result.
Preserve the alias predicate above its projection when the input contains a filter, join, limit, or CTE consumer. Apply the same protection in logical rewriting and physical postprocessing. CTE consumers need protection because their predicates can be pushed into producers whose filtering conditions are outside the consumer's input tree.
Ordinary column predicates and date aliases without a lower evaluation boundary retain predicate pushdown and partition pruning. Overflow on rows that actually participate in date arithmetic still raises an error.
Validation:
- FE: 14 tests passed across `DateArithmeticFilterPushDownTest`, `postprocess.PushDownFilterThroughProjectTest`, and `PushDownFilterThroughAggregationTest`. The new safe-pushdown tests failed before the boundary restriction was narrowed.
- BE: `VTimestampFunctionsTest.weeks_add_v2_boundary_test` and `weeks_add_v2_test` passed; BE arithmetic code is unchanged.
- Local regression: `test_date_expression_filter`, `doc_date_error`, `test_cte_filter_pushdown`, and `date_function_rewrite` passed after deploying the branch build. The new suite checks ORC/Parquet input, shared non-inlined CTEs, valid results, real overflow, NULL, and safe date-alias pruning to one of three partitions.
- The representative Jira SQL returned zero rows without an error on reconstructed 200-row ORC and Parquet input. Historical DLF object bytes were unavailable, so this is reconstructed-input validation.
- FE Checkstyle and `git diff --check` passed. These results cover the targeted local suites.
The four additional cases mentioned in Jira have different SQL shapes and have not been established as the same root cause; this PR addresses the confirmed representative failure.
### Release note
Fix erroneous date arithmetic overflow errors caused by predicate pushdown in filtered subqueries and CTEs.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Date arithmetic alias predicates preserve inner filtering boundaries; safe scan-only aliases remain eligible for pushdown.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) Preserve date arithmetic evaluation boundaries [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69073:
URL: https://github.com/apache/doris/pull/69073#issuecomment-6172264380
# FE UT Coverage Report
Increment line coverage ` 100.00% (8/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69073_1c762dd2a20caa63d2f4acbce18edd7c04c0af3e/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69073_1c762dd2a20caa63d2f4acbce18edd7c04c0af3e/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) Preserve date arithmetic evaluation boundaries [doris]
Posted by "OIiveirra (via GitHub)" <gi...@apache.org>.
OIiveirra commented on PR #69073:
URL: https://github.com/apache/doris/pull/69073#issuecomment-6168030722
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) Preserve date arithmetic evaluation boundaries [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69073:
URL: https://github.com/apache/doris/pull/69073#discussion_r4270768938
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -133,6 +140,16 @@ public class ExpressionUtils {
// folding the conjunct). We restrict MAX_MARK_SLOT_COUNT to 4 to bound this cost.
private static final int MAX_MARK_SLOT_COUNT = 4;
+ /**
+ * Preserve date arithmetic's evaluation domain above a lower condition or limit.
+ * A CTE consumer may also push a predicate into a producer whose conditions are outside this input tree.
+ */
+ public static boolean requiresDateArithmeticProtection(Expression expression, Plan input) {
+ return expression.containsType(DateAddSubMonotonic.class, DateCeilFloorMonotonic.class)
Review Comment:
[P2] Preserve partition pruning for safe zero-offset date aliases
Adding an unrelated inner WHERE to the new pruning case now loses pruning:
```text
Filter(d = DATE '2020-01-02')
Project(DAYS_ADD(dt, 0) AS d, id)
Filter(id > 0)
Scan(t partitioned by dt)
```
The marker plus lower Filter makes this helper return true, so the date equality stays above Project. PruneOlapScanPartition sees only id > 0 beside the scan and selects all three daily partitions instead of p2; the same shape can scan every partition of a large table. Previously alias substitution exposed DAYS_ADD(dt,0)=literal, which SimplifyArithmeticComparisonRule reduced to dt=DAYS_SUB(literal,0) and then dt=literal. Zero-day addition is safe for every DATE in the test table's 2020 ranges, so this extra restriction does not protect against overflow. Keep provably safe aliases eligible for pushdown, or derive an equivalent safe base-column predicate for pruning, and extend the regression case with this inner WHERE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) Preserve date arithmetic evaluation boundaries [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69073:
URL: https://github.com/apache/doris/pull/69073#issuecomment-6173894383
# FE Regression Coverage Report
Increment line coverage ` 87.50% (7/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69073_1c762dd2a20caa63d2f4acbce18edd7c04c0af3e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69073_1c762dd2a20caa63d2f4acbce18edd7c04c0af3e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) Preserve date arithmetic evaluation boundaries [doris]
Posted by "OIiveirra (via GitHub)" <gi...@apache.org>.
OIiveirra closed pull request #69073: [fix](nereids) Preserve date arithmetic evaluation boundaries
URL: https://github.com/apache/doris/pull/69073
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) Preserve date arithmetic evaluation boundaries [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69073:
URL: https://github.com/apache/doris/pull/69073#issuecomment-6167989617
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) Preserve date arithmetic evaluation boundaries [doris]
Posted by "OIiveirra (via GitHub)" <gi...@apache.org>.
OIiveirra commented on PR #69073:
URL: https://github.com/apache/doris/pull/69073#issuecomment-6184106346
Closing the 4.1 PR temporarily. The fix will be reworked and validated on master first, then picked back to 4.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) Preserve date arithmetic evaluation boundaries [doris]
Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #69073:
URL: https://github.com/apache/doris/pull/69073#issuecomment-6168039178
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org