You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2026/04/10 02:39:27 UTC
[PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
englefly opened a new pull request, #62304:
URL: https://github.com/apache/doris/pull/62304
### What problem does this PR solve?
Treat nullable fields as a combination of a nullable flag and data. When evaluating the `col IS NULL` predicate, use the NestedColumnPruning rule to prune the col field to col.NULL, thereby saving I/O on the data.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4317552504
# FE Regression Coverage Report
Increment line coverage ` 88.79% (95/107)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_f8310449fd5a0bd83cf347a7e80a657fb1277ad2_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_f8310449fd5a0bd83cf347a7e80a657fb1277ad2_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4411864931
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414233406
# FE UT Coverage Report
Increment line coverage ` 76.61% (131/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_8a43bbd701076d5edc3c81c31cb527861fe4040e/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_8a43bbd701076d5edc3c81c31cb527861fe4040e/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62304:
URL: https://github.com/apache/doris/pull/62304#discussion_r3097854572
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##########
@@ -286,6 +340,10 @@ private static Map<Integer, AccessPathInfo> pruneDataType(
});
Review Comment:
`stripNullSuffixPaths()` breaks mixed projection/filter cases. BE readers use `all_access_paths` to decide which nested children are materialized, and `predicate_access_paths` is only forwarded after that child has already been selected. For example, `select struct_element(s, 'data') from tbl where struct_element(s, 'city') is null` now becomes `all=[s.data]` and `predicate=[s.city.NULL]`; `StructFileColumnIterator::set_access_paths()` will skip `city` entirely and the filter will run on default-filled data. Please keep the NULL path in `allAccessPaths` for mixed-access cases, or teach the BE readers to treat predicate-only subpaths as readable.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java:
##########
@@ -72,7 +72,8 @@ public Map<Slot, List<CollectAccessPathResult>> collect(Plan root, StatementCont
private boolean shouldCollectAccessPath(Slot slot) {
return slot.getDataType() instanceof NestedColumnPrunable
|| slot.getDataType().isVariantType()
- || slot.getDataType().isStringLikeType();
+ || slot.getDataType().isStringLikeType()
+ || slot.nullable();
Review Comment:
This broadens the optimization to every nullable scalar, but the BE scalar reader path still does not honor `[col, NULL]`. For OLAP scans we eventually land in `FileColumnIterator::set_access_paths()`, and that default implementation ignores `all_access_paths` completely, so `INT/DATE/... IS NULL` still reads full data pages. The new explain-only cases for `int_col.NULL` therefore look optimized even though no I/O is saved. Please either keep this scoped to readers with actual `NULL_MAP_ONLY` support, or add the missing BE scalar-reader handling in the same PR.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##########
@@ -472,6 +556,17 @@ public boolean hasStringOffsetOnlyAccess() {
return type.isStringLikeType() && isStringOffsetOnly && !accessAll;
}
+ /** True when the column is accessed ONLY via IS NULL / IS NOT NULL,
+ * meaning the BE only needs to read the null flag, not the actual data. */
+ public boolean hasNullCheckOnlyAccess() {
+ if (isRoot) {
+ DataTypeAccessTree child = children.values().iterator().next();
+ return child.isNullCheckOnly && !child.accessAll
+ && !child.isStringOffsetOnly && !child.accessPartialChild;
+ }
+ return isNullCheckOnly && !accessAll && !isStringOffsetOnly && !accessPartialChild;
+ }
+
/** pruneCastType */
Review Comment:
For a top-level string predicate like `str_col IS NULL`, this never becomes `hasNullCheckOnlyAccess() == true`. By the time we hit this branch, `accessPartialChild` has already been set in the method prologue, so the root check at lines 561-567 still returns false. Because string slots take the dedicated branch earlier in `pruneDataType()` and `continue`, FE ends up attaching no access path at all, so the optimization does not trigger for top-level string columns. Could we avoid setting `accessPartialChild` for terminal `NULL` markers, or relax `hasNullCheckOnlyAccess()` for leaf nodes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4317382703
# FE UT Coverage Report
Increment line coverage ` 89.72% (96/107)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_f8310449fd5a0bd83cf347a7e80a657fb1277ad2/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_f8310449fd5a0bd83cf347a7e80a657fb1277ad2/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4417740756
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.61% (27790/37755) |
| Line Coverage | 57.56% (300874/522736) |
| Region Coverage | 54.84% (251062/457768) |
| Branch Coverage | 56.32% (108425/192506) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4420520779
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4411792953
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4412162899
<details>
<summary>TPC-DS: <b>Total hot run time: 170878 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f0cfe863011095f809f57d6db6e0a86c2de5ca1a, data reload: false
query5 4340 656 517 517
query6 324 215 199 199
query7 4273 574 344 344
query8 356 227 214 214
query9 8813 3970 4004 3970
query10 446 340 295 295
query11 5790 2427 2199 2199
query12 190 130 125 125
query13 1302 593 438 438
query14 5974 5391 5090 5090
query14_1 4420 4385 4379 4379
query15 212 203 188 188
query16 1007 462 461 461
query17 1159 800 635 635
query18 2464 495 366 366
query19 223 209 180 180
query20 142 137 133 133
query21 219 147 118 118
query22 13576 13967 14593 13967
query23 17268 16602 16180 16180
query23_1 16341 16328 16327 16327
query24 7471 1752 1395 1395
query24_1 1341 1342 1344 1342
query25 547 507 430 430
query26 1317 318 167 167
query27 2682 611 332 332
query28 4386 1970 1941 1941
query29 988 622 515 515
query30 296 237 193 193
query31 1091 1067 924 924
query32 78 72 68 68
query33 548 339 279 279
query34 1167 1130 641 641
query35 754 776 664 664
query36 1324 1347 1239 1239
query37 153 98 91 91
query38 3202 3128 3051 3051
query39 921 911 885 885
query39_1 869 867 874 867
query40 231 157 135 135
query41 63 59 58 58
query42 111 105 108 105
query43 323 320 281 281
query44
query45 208 208 195 195
query46 1096 1163 708 708
query47 2327 2301 2221 2221
query48 400 409 295 295
query49 628 516 432 432
query50 707 290 212 212
query51 4319 4248 4212 4212
query52 104 102 92 92
query53 267 272 201 201
query54 312 271 250 250
query55 93 92 87 87
query56 303 302 304 302
query57 1407 1408 1302 1302
query58 285 268 282 268
query59 1541 1630 1442 1442
query60 341 338 320 320
query61 160 158 156 156
query62 654 624 570 570
query63 241 198 201 198
query64 2436 817 672 672
query65
query66 1765 504 388 388
query67 30064 29973 29781 29781
query68
query69 459 336 296 296
query70 1016 982 903 903
query71 308 273 274 273
query72 2960 2737 2461 2461
query73 855 737 402 402
query74 5086 4920 4726 4726
query75 2770 2656 2339 2339
query76 2294 1128 781 781
query77 405 428 342 342
query78 12925 12996 12255 12255
query79 1493 991 742 742
query80 1371 598 494 494
query81 504 278 239 239
query82 1315 158 122 122
query83 356 274 245 245
query84 271 138 108 108
query85 898 524 445 445
query86 456 330 327 327
query87 3434 3356 3214 3214
query88 3490 2669 2642 2642
query89 439 383 336 336
query90 1956 176 181 176
query91 180 172 140 140
query92 79 75 71 71
query93 960 961 561 561
query94 716 362 307 307
query95 663 383 366 366
query96 1022 776 342 342
query97 2712 2669 2576 2576
query98 247 235 232 232
query99 1140 1107 987 987
Total cold run time: 254316 ms
Total hot run time: 170878 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4412008514
<details>
<summary>TPC-H: <b>Total hot run time: 29711 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f0cfe863011095f809f57d6db6e0a86c2de5ca1a, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17724 3930 4084 3930
q2 q3 10727 863 605 605
q4 4656 460 354 354
q5 7450 1339 1151 1151
q6 196 165 145 145
q7 933 930 748 748
q8 9662 1378 1315 1315
q9 6450 5369 5349 5349
q10 6335 2069 1827 1827
q11 474 264 255 255
q12 682 434 294 294
q13 18165 3286 2741 2741
q14 291 283 263 263
q15 q16 906 880 789 789
q17 1060 1087 742 742
q18 6503 5701 5643 5643
q19 1712 1271 988 988
q20 511 393 262 262
q21 4952 2352 1968 1968
q22 477 389 342 342
Total cold run time: 99866 ms
Total hot run time: 29711 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4692 4540 4603 4540
q2 q3 4645 4793 4235 4235
q4 2197 2173 1406 1406
q5 5017 4992 5281 4992
q6 200 173 143 143
q7 2001 1808 1603 1603
q8 3368 3099 3101 3099
q9 8510 8688 8507 8507
q10 4498 4523 4263 4263
q11 598 416 391 391
q12 726 745 562 562
q13 3246 3586 3000 3000
q14 307 307 276 276
q15 q16 768 781 706 706
q17 1387 1341 1306 1306
q18 7947 7156 7121 7121
q19 1177 1151 1168 1151
q20 2231 2228 1988 1988
q21 6111 5293 4875 4875
q22 528 483 408 408
Total cold run time: 60154 ms
Total hot run time: 54572 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4395480024
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4402499224
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #62304:
URL: https://github.com/apache/doris/pull/62304#discussion_r3214105097
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java:
##########
@@ -506,14 +546,35 @@ public Void visitArraySortBy(ArraySortBy arraySortBy, CollectorContext context)
return visit(arraySortBy, context);
}
- // @Override
- // public Void visitIsNull(IsNull isNull, CollectorContext context) {
- // if (context.accessPathBuilder.isEmpty()) {
- // context.setType(ColumnAccessPathType.META);
- // return continueCollectAccessPath(isNull.child(), context);
- // }
- // return visit(isNull, context);
- // }
+ @Override
+ public Void visitIsNull(IsNull isNull, CollectorContext context) {
+ Expression arg = isNull.child();
+ // Skip variant sub-column paths (v['k'] IS NULL): the sub-column path is already baked
+ // into the SlotReference, so null-only access doesn't apply the same way.
+ if (arg instanceof SlotReference && ((SlotReference) arg).hasSubColPath()) {
+ return visit(isNull, context);
+ }
+ // Optimize IS NULL on nullable expressions: create a context with NULL suffix to indicate
+ // only the null flag is needed. Works for top-level columns (col IS NULL → [col, NULL])
+ // and nested access (struct_element(s, 'city') IS NULL → [s, city, NULL]).
+ // For unrecognized expressions, the default visitor resets context, safely discarding NULL.
+ if (arg.nullable() && context.accessPathBuilder.isEmpty()) {
+ CollectorContext nullContext =
+ new CollectorContext(context.statementContext, context.bottomFilter);
+ nullContext.accessPathBuilder.addSuffix(AccessPathInfo.ACCESS_NULL);
+ return continueCollectAccessPath(arg, nullContext);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414072128
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4428630883
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.70% (27800/37718) |
| Line Coverage | 57.55% (300824/522712) |
| Region Coverage | 54.72% (250766/458245) |
| Branch Coverage | 56.26% (108508/192861) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4427157915
run beut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4265080344
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4411793167
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4411865329
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414121517
<details>
<summary>TPC-H: <b>Total hot run time: 29667 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a43bbd701076d5edc3c81c31cb527861fe4040e, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17734 3882 3857 3857
q2 q3 10716 900 620 620
q4 4671 464 362 362
q5 7501 1368 1167 1167
q6 204 168 143 143
q7 953 960 789 789
q8 9519 1431 1340 1340
q9 6089 5426 5369 5369
q10 6297 2087 1822 1822
q11 481 275 273 273
q12 699 431 311 311
q13 18164 3326 2744 2744
q14 295 292 266 266
q15 q16 924 860 788 788
q17 1029 1081 689 689
q18 6471 5707 5575 5575
q19 1178 1242 1114 1114
q20 501 403 263 263
q21 4680 2289 1869 1869
q22 419 354 306 306
Total cold run time: 98525 ms
Total hot run time: 29667 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4191 4125 4230 4125
q2 q3 4628 4776 4157 4157
q4 2090 2171 1412 1412
q5 4968 5039 5272 5039
q6 185 162 136 136
q7 2028 1792 1742 1742
q8 3484 3213 3172 3172
q9 8485 8509 8316 8316
q10 4484 4503 4261 4261
q11 584 422 418 418
q12 717 772 543 543
q13 3243 3590 2941 2941
q14 307 306 266 266
q15 q16 767 780 678 678
q17 1347 1322 1290 1290
q18 8041 7164 7107 7107
q19 1164 1155 1183 1155
q20 2268 2232 1959 1959
q21 6100 5408 4905 4905
q22 553 540 423 423
Total cold run time: 59634 ms
Total hot run time: 54045 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414138300
<details>
<summary>TPC-DS: <b>Total hot run time: 171552 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a43bbd701076d5edc3c81c31cb527861fe4040e, data reload: false
query5 4327 668 517 517
query6 348 248 221 221
query7 4239 583 312 312
query8 336 244 238 238
query9 8829 4122 4041 4041
query10 501 355 306 306
query11 6085 2393 2211 2211
query12 194 133 132 132
query13 1288 614 464 464
query14 6885 5419 5104 5104
query14_1 4430 4437 4370 4370
query15 213 210 189 189
query16 968 461 445 445
query17 1388 782 658 658
query18 2742 511 366 366
query19 322 217 170 170
query20 141 136 130 130
query21 217 143 119 119
query22 13621 13586 13340 13340
query23 17198 16365 16591 16365
query23_1 16281 16310 16411 16310
query24 7963 1855 1392 1392
query24_1 1465 1401 1421 1401
query25 589 509 461 461
query26 1323 348 190 190
query27 2851 639 362 362
query28 4360 2012 1980 1980
query29 1017 653 531 531
query30 301 233 197 197
query31 1125 1079 941 941
query32 81 75 73 73
query33 524 341 287 287
query34 1122 1137 643 643
query35 768 791 671 671
query36 1312 1274 1135 1135
query37 149 100 89 89
query38 3227 3173 3071 3071
query39 936 935 929 929
query39_1 885 867 868 867
query40 234 153 137 137
query41 63 62 62 62
query42 112 106 110 106
query43 333 332 288 288
query44
query45 210 207 199 199
query46 1129 1212 731 731
query47 2294 2271 2208 2208
query48 417 429 277 277
query49 631 531 459 459
query50 756 307 236 236
query51 4416 4337 4349 4337
query52 110 109 99 99
query53 263 286 211 211
query54 312 277 254 254
query55 96 94 84 84
query56 309 300 322 300
query57 1455 1408 1268 1268
query58 290 273 266 266
query59 1606 1651 1463 1463
query60 343 342 324 324
query61 151 157 155 155
query62 664 617 569 569
query63 246 205 205 205
query64 2318 835 698 698
query65
query66 1667 508 407 407
query67 30274 30090 29911 29911
query68
query69 469 333 304 304
query70 1006 1027 966 966
query71 310 282 275 275
query72 2933 2679 2399 2399
query73 836 765 423 423
query74 5072 4914 4758 4758
query75 2758 2673 2344 2344
query76 2289 1155 818 818
query77 417 433 340 340
query78 12859 12984 12502 12502
query79 1497 1001 757 757
query80 785 600 499 499
query81 469 285 239 239
query82 1255 157 121 121
query83 363 283 256 256
query84 264 144 109 109
query85 885 483 449 449
query86 414 352 313 313
query87 3443 3348 3221 3221
query88 3593 2661 2649 2649
query89 440 385 340 340
query90 1796 179 188 179
query91 178 169 136 136
query92 79 79 74 74
query93 994 976 566 566
query94 596 333 295 295
query95 665 448 347 347
query96 1008 763 351 351
query97 2687 2683 2612 2612
query98 239 229 227 227
query99 1095 1115 981 981
Total cold run time: 255735 ms
Total hot run time: 171552 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4417029669
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.57% (20652/38551) |
| Line Coverage | 37.19% (194906/524109) |
| Region Coverage | 33.50% (151884/453348) |
| Branch Coverage | 34.55% (66264/191778) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4418099162
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4317437593
OpenCode automated review failed and did not complete.
Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/24916998608
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4314912686
# FE Regression Coverage Report
Increment line coverage ` 88.79% (95/107)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_66069bc6fa759d2a07384e6438aa9a765d229066_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_66069bc6fa759d2a07384e6438aa9a765d229066_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #62304:
URL: https://github.com/apache/doris/pull/62304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4265079868
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62304:
URL: https://github.com/apache/doris/pull/62304#discussion_r3212691845
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##########
@@ -246,7 +301,22 @@ private static Map<Integer, AccessPathInfo> pruneDataType(
continue;
}
+ // Null-check-only access (e.g. col IS NULL / col IS NOT NULL): type stays unchanged,
+ // but we must send the [col, NULL] access path to BE so it only reads the null flag.
+ if (accessTree.hasNullCheckOnlyAccess()) {
+ if (skipDataSkippingOnlyAccessPath) {
Review Comment:
This MV-fragment skip only catches top-level `col IS NULL`/`IS NOT NULL` cases. For a nested predicate like `struct_element(s, 'city') is not null`, `hasNullCheckOnlyAccess()` is false because the root `s` has `accessPartialChild`, so the code falls through and still emits `[s.city.NULL]` plus a pruned `struct<city:...>` type even when `skipDataSkippingOnlyAccessPath` is true. That can reproduce the same MV temporary-fragment bug this flag is trying to avoid: the temporary rewritten fragment only needs the nested null flag for filtering, but the final MV rewritten plan may still need the full complex slot or another child. This is distinct from the existing parent-NULL/allAccessPath threads; it is the MV-rewrite skip path not covering nested NULL-only access. Please either suppress nested data-skipping-only access paths under this flag as well, or add a nested MV rewrite regression that proves the final plan cannot reuse a pruned complex slot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414346844
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4416929210
<details>
<summary>TPC-DS: <b>Total hot run time: 170101 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016, data reload: false
query5 4338 653 524 524
query6 327 216 200 200
query7 4275 558 323 323
query8 326 231 221 221
query9 8826 4183 4126 4126
query10 467 356 305 305
query11 5818 2411 2255 2255
query12 185 131 134 131
query13 1278 622 446 446
query14 6269 5381 5123 5123
query14_1 4417 4397 4349 4349
query15 215 204 184 184
query16 1032 472 444 444
query17 1150 791 617 617
query18 2743 478 340 340
query19 213 198 157 157
query20 138 131 131 131
query21 212 135 113 113
query22 13649 13646 13371 13371
query23 17165 16357 15907 15907
query23_1 16137 16164 16210 16164
query24 7393 1738 1348 1348
query24_1 1357 1373 1343 1343
query25 551 496 442 442
query26 1311 304 167 167
query27 2769 583 333 333
query28 4409 2018 1967 1967
query29 1002 639 512 512
query30 311 234 198 198
query31 1127 1064 928 928
query32 86 73 75 73
query33 530 349 286 286
query34 1164 1106 635 635
query35 768 789 666 666
query36 1330 1307 1211 1211
query37 158 108 92 92
query38 3198 3157 3051 3051
query39 974 971 902 902
query39_1 904 912 934 912
query40 238 156 139 139
query41 68 64 61 61
query42 115 108 111 108
query43 328 341 303 303
query44
query45 206 203 195 195
query46 1074 1217 789 789
query47 2332 2355 2151 2151
query48 413 439 306 306
query49 676 547 449 449
query50 719 291 223 223
query51 4287 4303 4306 4303
query52 107 108 95 95
query53 262 291 214 214
query54 333 298 275 275
query55 103 90 86 86
query56 318 319 322 319
query57 1402 1397 1305 1305
query58 305 281 265 265
query59 1577 1622 1430 1430
query60 359 351 332 332
query61 180 175 179 175
query62 687 629 559 559
query63 249 200 210 200
query64 2425 860 735 735
query65
query66 1699 538 417 417
query67 29965 29300 29832 29300
query68
query69 468 330 294 294
query70 960 1010 1002 1002
query71 310 276 270 270
query72 2940 2758 2470 2470
query73 862 779 445 445
query74 5087 4884 4725 4725
query75 2798 2686 2344 2344
query76 2280 1137 742 742
query77 420 417 352 352
query78 12950 12996 12386 12386
query79 1503 987 773 773
query80 1410 594 493 493
query81 537 281 240 240
query82 1121 157 123 123
query83 327 292 254 254
query84 265 148 116 116
query85 931 518 454 454
query86 463 322 330 322
query87 3442 3367 3230 3230
query88 3574 2767 2704 2704
query89 458 377 338 338
query90 1957 179 183 179
query91 177 174 141 141
query92 82 78 74 74
query93 1062 966 564 564
query94 717 343 295 295
query95 687 362 351 351
query96 997 781 317 317
query97 2719 2671 2566 2566
query98 243 233 244 233
query99 1150 1120 1007 1007
Total cold run time: 255066 ms
Total hot run time: 170101 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414410554
<details>
<summary>TPC-DS: <b>Total hot run time: 171966 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a43bbd701076d5edc3c81c31cb527861fe4040e, data reload: false
query5 4339 654 527 527
query6 335 237 215 215
query7 4226 552 315 315
query8 341 246 231 231
query9 8859 4110 4091 4091
query10 447 347 309 309
query11 5763 2390 2211 2211
query12 183 135 131 131
query13 1281 633 440 440
query14 6744 5334 5080 5080
query14_1 4342 4378 4344 4344
query15 217 205 189 189
query16 1012 457 453 453
query17 1157 774 649 649
query18 2777 500 372 372
query19 233 212 172 172
query20 140 135 133 133
query21 216 138 120 120
query22 13569 13592 13357 13357
query23 17179 16434 16479 16434
query23_1 16409 16279 16320 16279
query24 7870 1819 1416 1416
query24_1 1416 1447 1414 1414
query25 614 570 497 497
query26 1411 373 179 179
query27 3154 618 345 345
query28 4451 2030 1960 1960
query29 1057 657 527 527
query30 317 248 191 191
query31 1111 1071 932 932
query32 84 67 71 67
query33 546 344 298 298
query34 1179 1164 609 609
query35 766 784 680 680
query36 1361 1322 1137 1137
query37 147 98 84 84
query38 3158 3109 3106 3106
query39 917 915 891 891
query39_1 859 869 875 869
query40 234 159 131 131
query41 62 64 57 57
query42 109 110 107 107
query43 322 323 277 277
query44
query45 205 196 188 188
query46 1083 1198 728 728
query47 2359 2297 2284 2284
query48 415 438 286 286
query49 640 531 437 437
query50 730 312 223 223
query51 4317 4306 4309 4306
query52 108 104 92 92
query53 253 297 211 211
query54 312 268 256 256
query55 95 100 83 83
query56 311 338 323 323
query57 1486 1445 1353 1353
query58 301 271 274 271
query59 1633 1710 1458 1458
query60 360 349 335 335
query61 147 154 161 154
query62 681 633 562 562
query63 247 206 208 206
query64 2348 829 677 677
query65
query66 1694 529 403 403
query67 30038 29964 29869 29869
query68
query69 455 354 302 302
query70 990 978 1000 978
query71 306 283 276 276
query72 2944 2665 2450 2450
query73 823 759 445 445
query74 5088 4873 4791 4791
query75 2811 2650 2320 2320
query76 2308 1138 763 763
query77 414 406 348 348
query78 12918 13004 12374 12374
query79 1518 1018 717 717
query80 1384 603 488 488
query81 521 292 240 240
query82 1075 165 119 119
query83 353 278 256 256
query84 274 139 118 118
query85 961 516 447 447
query86 464 321 306 306
query87 3430 3350 3244 3244
query88 3585 2695 2683 2683
query89 438 380 339 339
query90 1968 179 175 175
query91 191 164 143 143
query92 76 76 68 68
query93 1049 986 592 592
query94 724 305 304 304
query95 675 369 344 344
query96 1047 812 351 351
query97 2729 2720 2620 2620
query98 242 235 236 235
query99 1123 1098 984 984
Total cold run time: 256338 ms
Total hot run time: 171966 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4417370563
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4421153907
<details>
<summary>TPC-DS: <b>Total hot run time: 170638 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 83d42d9aa645228fb4135a059ce26a489db011c4, data reload: false
query5 4307 652 524 524
query6 334 229 193 193
query7 4214 560 305 305
query8 337 230 214 214
query9 8815 3976 3964 3964
query10 430 332 304 304
query11 5824 2395 2221 2221
query12 194 138 133 133
query13 1301 613 450 450
query14 5950 5367 5090 5090
query14_1 4364 4395 4370 4370
query15 215 202 182 182
query16 1011 448 432 432
query17 1164 790 634 634
query18 2507 496 378 378
query19 215 212 170 170
query20 142 139 130 130
query21 230 139 120 120
query22 13638 13692 13377 13377
query23 17316 16355 16000 16000
query23_1 16250 16197 16177 16177
query24 7463 1750 1350 1350
query24_1 1367 1367 1341 1341
query25 594 519 480 480
query26 1309 293 172 172
query27 2715 620 345 345
query28 4426 1958 1935 1935
query29 1014 640 515 515
query30 309 240 199 199
query31 1107 1064 942 942
query32 85 78 73 73
query33 535 354 301 301
query34 1163 1102 656 656
query35 767 790 685 685
query36 1301 1333 1137 1137
query37 157 97 86 86
query38 3228 3077 3056 3056
query39 926 936 891 891
query39_1 888 874 873 873
query40 241 160 135 135
query41 66 62 63 62
query42 115 111 109 109
query43 326 327 290 290
query44
query45 213 199 192 192
query46 1085 1189 706 706
query47 2376 2403 2278 2278
query48 406 420 291 291
query49 663 549 469 469
query50 715 303 232 232
query51 4399 4378 4245 4245
query52 113 109 95 95
query53 251 280 205 205
query54 311 272 248 248
query55 92 88 83 83
query56 305 319 328 319
query57 1404 1406 1326 1326
query58 311 283 267 267
query59 1542 1605 1390 1390
query60 339 338 327 327
query61 162 156 158 156
query62 668 616 561 561
query63 240 205 217 205
query64 2456 837 685 685
query65
query66 1764 508 384 384
query67 30039 29366 29794 29366
query68
query69 476 336 301 301
query70 998 1051 995 995
query71 305 279 269 269
query72 2917 2698 2404 2404
query73 824 771 411 411
query74 5035 4903 4743 4743
query75 2786 2676 2341 2341
query76 2280 1123 774 774
query77 425 428 363 363
query78 12951 12964 12337 12337
query79 1493 984 749 749
query80 966 620 511 511
query81 503 316 244 244
query82 1332 162 127 127
query83 362 275 244 244
query84 255 145 115 115
query85 908 508 435 435
query86 422 359 330 330
query87 3414 3344 3219 3219
query88 3521 2666 2645 2645
query89 450 387 347 347
query90 1813 177 185 177
query91 173 170 137 137
query92 82 78 74 74
query93 955 948 566 566
query94 617 350 298 298
query95 659 376 427 376
query96 1011 756 360 360
query97 2718 2719 2574 2574
query98 248 236 233 233
query99 1114 1122 993 993
Total cold run time: 253674 ms
Total hot run time: 170638 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4425368319
# FE Regression Coverage Report
Increment line coverage ` 79.79% (150/188)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4384723759
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4393399696
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4340061645
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4415595086
<details>
<summary>TPC-H: <b>Total hot run time: 29660 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 54a413523493235f44674286e28354973dfab413, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17596 3973 3984 3973
q2 q3 10716 868 604 604
q4 4660 464 353 353
q5 7441 1339 1171 1171
q6 199 181 151 151
q7 929 982 744 744
q8 9331 1464 1237 1237
q9 5668 5386 5303 5303
q10 6245 2062 1799 1799
q11 468 268 255 255
q12 637 439 305 305
q13 18118 3333 2702 2702
q14 294 291 271 271
q15 q16 885 864 796 796
q17 1001 1062 735 735
q18 6405 5738 5603 5603
q19 1253 1226 1110 1110
q20 528 395 256 256
q21 4553 2465 1963 1963
q22 492 381 329 329
Total cold run time: 97419 ms
Total hot run time: 29660 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5066 4786 5049 4786
q2 q3 4658 4805 4241 4241
q4 2312 2240 1451 1451
q5 4999 4997 5199 4997
q6 203 184 141 141
q7 2104 1879 1622 1622
q8 3323 3112 3123 3112
q9 8452 8435 8494 8435
q10 4568 4549 4291 4291
q11 613 429 429 429
q12 704 757 541 541
q13 3222 3608 2898 2898
q14 300 299 298 298
q15 q16 928 774 723 723
q17 1373 1367 1293 1293
q18 8049 7186 7210 7186
q19 1164 1150 1191 1150
q20 2287 2270 1963 1963
q21 6358 5628 5013 5013
q22 582 504 437 437
Total cold run time: 61265 ms
Total hot run time: 55007 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4415749628
# FE UT Coverage Report
Increment line coverage ` 76.61% (131/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_54a413523493235f44674286e28354973dfab413/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_54a413523493235f44674286e28354973dfab413/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62304:
URL: https://github.com/apache/doris/pull/62304#discussion_r3205665945
##########
task.md:
##########
@@ -0,0 +1,15 @@
+当前分支实现了 is-null 谓词的优化.
Review Comment:
This looks like a local scratch note describing the debugging task and stack trace, not a Doris source/test/doc change. Please remove it from the PR before merging.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##########
@@ -480,6 +541,60 @@ private List<List<String>> getSupplementalPaths() {
}
Review Comment:
This parent-`NULL` path cannot be kept in `allAccessPaths` together with child data paths with the current BE contract. For `select struct_element(struct_col, 'city') from ncp_tbl where struct_col is null`, FE now sends `all=[struct_col.NULL, struct_col.city]`. In BE, `StructFileColumnIterator::set_access_paths()` strips the root name, sees `NULL` in `sub_all_access_paths`, `_check_and_set_meta_read_mode()` switches the whole struct iterator to `NULL_MAP_ONLY`, and lines 1562-1568 skip all sub-column iterators before `city` is routed. The projected `city` value is then default-filled instead of read. This is distinct from the existing predicate-only child-path thread: here the child path is present in `allAccessPaths`, but the parent `NULL` meta mode takes precedence and suppresses it. Please either avoid emitting parent `.NULL` in `allAccessPaths` when any real child path is also required, or update the BE iterator logic so `NULL_MAP_ONLY` only applies when the null-map path is t
he only required access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4416809255
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4417046766
# FE UT Coverage Report
Increment line coverage ` 76.61% (131/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4416896005
<details>
<summary>TPC-H: <b>Total hot run time: 29459 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17809 3847 3910 3847
q2 q3 10725 858 591 591
q4 4659 460 336 336
q5 7447 1323 1146 1146
q6 193 170 139 139
q7 890 948 754 754
q8 9304 1409 1308 1308
q9 5661 5349 5313 5313
q10 6318 2086 1810 1810
q11 476 262 258 258
q12 661 418 290 290
q13 18188 3253 2713 2713
q14 285 282 264 264
q15 q16 905 880 785 785
q17 986 1099 778 778
q18 6443 5685 5581 5581
q19 1295 1218 1053 1053
q20 513 396 252 252
q21 4826 2370 1913 1913
q22 475 382 328 328
Total cold run time: 98059 ms
Total hot run time: 29459 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4657 4540 4602 4540
q2 q3 4637 4776 4180 4180
q4 2113 2195 1399 1399
q5 4967 5048 5290 5048
q6 195 175 140 140
q7 2075 1794 1629 1629
q8 3339 3132 3118 3118
q9 8504 8502 8439 8439
q10 4481 4466 4227 4227
q11 618 427 401 401
q12 709 773 530 530
q13 3211 3653 2972 2972
q14 311 318 276 276
q15 q16 778 769 693 693
q17 1350 1357 1299 1299
q18 7969 7205 7074 7074
q19 1150 1169 1142 1142
q20 2214 2241 1977 1977
q21 6198 5458 4822 4822
q22 535 480 402 402
Total cold run time: 60011 ms
Total hot run time: 54308 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4412031802
<details>
<summary>TPC-DS: <b>Total hot run time: 172661 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f0cfe863011095f809f57d6db6e0a86c2de5ca1a, data reload: false
query5 4310 687 526 526
query6 332 246 225 225
query7 4245 579 328 328
query8 334 240 222 222
query9 8812 4140 4080 4080
query10 468 350 313 313
query11 5908 2607 2258 2258
query12 190 134 133 133
query13 1299 632 434 434
query14 6862 5398 5111 5111
query14_1 4383 4384 4332 4332
query15 210 207 182 182
query16 1032 464 433 433
query17 1194 785 623 623
query18 2735 488 355 355
query19 219 200 173 173
query20 149 134 133 133
query21 217 138 120 120
query22 13618 13607 13377 13377
query23 17249 16547 16038 16038
query23_1 16187 16074 16145 16074
query24 7453 1764 1357 1357
query24_1 1387 1345 1403 1345
query25 553 521 482 482
query26 1319 322 179 179
query27 2709 598 351 351
query28 4408 2014 1989 1989
query29 1022 695 561 561
query30 310 247 205 205
query31 1133 1068 958 958
query32 92 81 77 77
query33 547 372 318 318
query34 1169 1142 673 673
query35 771 798 695 695
query36 1292 1337 1101 1101
query37 162 112 99 99
query38 3255 3181 3087 3087
query39 943 967 936 936
query39_1 906 919 912 912
query40 248 164 157 157
query41 76 70 68 68
query42 123 123 127 123
query43 350 348 298 298
query44
query45 222 209 197 197
query46 1170 1286 758 758
query47 2281 2298 2149 2149
query48 434 396 316 316
query49 650 558 445 445
query50 737 294 229 229
query51 4314 4264 4208 4208
query52 110 112 103 103
query53 265 297 209 209
query54 349 289 274 274
query55 98 95 89 89
query56 322 337 328 328
query57 1412 1385 1337 1337
query58 315 287 285 285
query59 1592 1643 1407 1407
query60 366 366 343 343
query61 189 186 185 185
query62 678 629 569 569
query63 254 214 210 210
query64 2414 829 680 680
query65
query66 1699 513 408 408
query67 30426 30031 30033 30031
query68
query69 480 344 307 307
query70 1063 1044 1030 1030
query71 324 283 272 272
query72 3008 2773 2451 2451
query73 855 787 420 420
query74 5134 4918 4796 4796
query75 2801 2671 2347 2347
query76 2308 1141 799 799
query77 427 433 352 352
query78 12986 13077 12422 12422
query79 1514 999 764 764
query80 843 601 489 489
query81 478 281 244 244
query82 1300 171 127 127
query83 387 284 262 262
query84 261 148 112 112
query85 934 527 453 453
query86 434 331 330 330
query87 3443 3363 3230 3230
query88 3606 2741 2675 2675
query89 449 389 350 350
query90 1781 186 182 182
query91 180 172 145 145
query92 82 76 75 75
query93 971 973 593 593
query94 614 352 313 313
query95 672 469 362 362
query96 1079 762 338 338
query97 2665 2746 2598 2598
query98 239 231 236 231
query99 1118 1088 969 969
Total cold run time: 255902 ms
Total hot run time: 172661 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4426291155
run beut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4386088903
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4396171829
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4221088978
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4402499671
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4418786719
# FE Regression Coverage Report
Increment line coverage ` 37.88% (136/359)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4422340660
# FE UT Coverage Report
Increment line coverage ` 76.60% (144/188)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414393146
<details>
<summary>TPC-H: <b>Total hot run time: 29345 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a43bbd701076d5edc3c81c31cb527861fe4040e, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17809 3876 3847 3847
q2 q3 10739 870 601 601
q4 4656 449 358 358
q5 7465 1331 1133 1133
q6 190 168 137 137
q7 943 933 770 770
q8 9297 1359 1209 1209
q9 5661 5408 5290 5290
q10 6298 2069 1802 1802
q11 489 273 262 262
q12 691 414 303 303
q13 18174 3339 2713 2713
q14 299 281 260 260
q15 q16 901 863 797 797
q17 1011 1085 712 712
q18 6441 5688 5595 5595
q19 1309 1289 1114 1114
q20 507 391 263 263
q21 4548 2450 1864 1864
q22 426 353 315 315
Total cold run time: 97854 ms
Total hot run time: 29345 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4181 4096 4067 4067
q2 q3 4614 4740 4200 4200
q4 2104 2171 1390 1390
q5 4968 4956 5246 4956
q6 185 164 131 131
q7 2038 1776 1909 1776
q8 3437 3156 3152 3152
q9 8523 8414 8495 8414
q10 4454 4506 4262 4262
q11 590 436 399 399
q12 733 744 534 534
q13 3153 3575 2954 2954
q14 294 316 269 269
q15 q16 761 783 710 710
q17 1352 1312 1251 1251
q18 7974 7032 7189 7032
q19 1184 1154 1197 1154
q20 2268 2288 1963 1963
q21 6117 5449 4934 4934
q22 564 511 426 426
Total cold run time: 59494 ms
Total hot run time: 53974 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4415559999
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4418712151
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.62% (27794/37755) |
| Line Coverage | 57.57% (300931/522736) |
| Region Coverage | 54.82% (250955/457768) |
| Branch Coverage | 56.32% (108414/192506) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4313938810
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4314707800
OpenCode automated review failed and did not complete.
Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/24894832326
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4221786883
# FE UT Coverage Report
Increment line coverage ` 88.89% (72/81)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_5f535837cb9a5835a1b4e7bc310e3e91c260a3a6/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_5f535837cb9a5835a1b4e7bc310e3e91c260a3a6/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4219667555
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4265546186
# FE UT Coverage Report
Increment line coverage ` 80.58% (83/103)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_eee072014d893e3257b248f16a29d6dba02a4429/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_eee072014d893e3257b248f16a29d6dba02a4429/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4393406856
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4411726102
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4428591768
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/83d42d9aa645228fb4135a059ce26a489db011c4_83d42d9aa645228fb4135a059ce26a489db011c4/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/83d42d9aa645228fb4135a059ce26a489db011c4_83d42d9aa645228fb4135a059ce26a489db011c4/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.57% (20634/38519) |
| Line Coverage | 37.19% (194919/524097) |
| Region Coverage | 33.57% (152342/453769) |
| Branch Coverage | 34.58% (66432/192121) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4415617975
<details>
<summary>TPC-DS: <b>Total hot run time: 170728 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 54a413523493235f44674286e28354973dfab413, data reload: false
query5 4329 673 538 538
query6 353 230 214 214
query7 4227 576 327 327
query8 333 240 224 224
query9 8848 4044 4002 4002
query10 472 347 308 308
query11 5779 2462 2214 2214
query12 185 132 128 128
query13 1271 647 455 455
query14 6095 5366 5050 5050
query14_1 4372 4402 4340 4340
query15 219 211 196 196
query16 983 493 375 375
query17 1144 786 647 647
query18 2513 523 380 380
query19 226 208 178 178
query20 144 136 134 134
query21 220 138 119 119
query22 13595 13853 14597 13853
query23 17418 16528 16071 16071
query23_1 16250 16307 16165 16165
query24 7405 1800 1325 1325
query24_1 1336 1342 1347 1342
query25 570 489 425 425
query26 1324 323 170 170
query27 2731 610 353 353
query28 4377 1940 1960 1940
query29 987 626 513 513
query30 304 226 191 191
query31 1099 1063 931 931
query32 83 70 71 70
query33 540 345 281 281
query34 1183 1152 639 639
query35 783 796 668 668
query36 1318 1329 1116 1116
query37 151 110 84 84
query38 3215 3137 3055 3055
query39 931 923 879 879
query39_1 863 883 868 868
query40 233 152 137 137
query41 62 60 58 58
query42 109 107 106 106
query43 335 328 284 284
query44
query45 213 199 193 193
query46 1092 1254 727 727
query47 2245 2255 2187 2187
query48 396 424 294 294
query49 655 511 427 427
query50 703 288 227 227
query51 4331 4335 4163 4163
query52 108 108 95 95
query53 253 281 206 206
query54 310 272 251 251
query55 97 88 84 84
query56 305 321 304 304
query57 1416 1405 1318 1318
query58 287 274 274 274
query59 1564 1617 1409 1409
query60 352 326 314 314
query61 150 160 189 160
query62 666 620 530 530
query63 241 216 208 208
query64 2440 809 671 671
query65
query66 1751 519 384 384
query67 29973 30002 29716 29716
query68
query69 459 334 300 300
query70 1018 981 972 972
query71 311 270 279 270
query72 2953 2697 2424 2424
query73 817 790 432 432
query74 5090 4890 4766 4766
query75 2834 2690 2331 2331
query76 2292 1133 792 792
query77 412 419 344 344
query78 13021 13011 12314 12314
query79 1491 1022 762 762
query80 1399 568 495 495
query81 539 280 233 233
query82 1232 156 124 124
query83 355 274 249 249
query84 256 137 117 117
query85 929 517 441 441
query86 455 322 336 322
query87 3427 3331 3236 3236
query88 3585 2704 2692 2692
query89 437 388 337 337
query90 1949 181 181 181
query91 177 170 139 139
query92 79 77 72 72
query93 955 964 567 567
query94 734 358 289 289
query95 672 376 344 344
query96 1030 754 342 342
query97 2675 2686 2582 2582
query98 238 229 235 229
query99 1151 1116 990 990
Total cold run time: 254313 ms
Total hot run time: 170728 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4432231933
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4317073497
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4317108443
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4265945038
# FE Regression Coverage Report
Increment line coverage ` 77.67% (80/103)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_eee072014d893e3257b248f16a29d6dba02a4429_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_eee072014d893e3257b248f16a29d6dba02a4429_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62304:
URL: https://github.com/apache/doris/pull/62304#discussion_r3212735833
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java:
##########
@@ -506,14 +546,35 @@ public Void visitArraySortBy(ArraySortBy arraySortBy, CollectorContext context)
return visit(arraySortBy, context);
}
- // @Override
- // public Void visitIsNull(IsNull isNull, CollectorContext context) {
- // if (context.accessPathBuilder.isEmpty()) {
- // context.setType(ColumnAccessPathType.META);
- // return continueCollectAccessPath(isNull.child(), context);
- // }
- // return visit(isNull, context);
- // }
+ @Override
+ public Void visitIsNull(IsNull isNull, CollectorContext context) {
+ Expression arg = isNull.child();
+ // Skip variant sub-column paths (v['k'] IS NULL): the sub-column path is already baked
+ // into the SlotReference, so null-only access doesn't apply the same way.
+ if (arg instanceof SlotReference && ((SlotReference) arg).hasSubColPath()) {
+ return visit(isNull, context);
+ }
+ // Optimize IS NULL on nullable expressions: create a context with NULL suffix to indicate
+ // only the null flag is needed. Works for top-level columns (col IS NULL → [col, NULL])
+ // and nested access (struct_element(s, 'city') IS NULL → [s, city, NULL]).
+ // For unrecognized expressions, the default visitor resets context, safely discarding NULL.
+ if (arg.nullable() && context.accessPathBuilder.isEmpty()) {
+ CollectorContext nullContext =
+ new CollectorContext(context.statementContext, context.bottomFilter);
+ nullContext.accessPathBuilder.addSuffix(AccessPathInfo.ACCESS_NULL);
+ return continueCollectAccessPath(arg, nullContext);
Review Comment:
This NULL suffix is also propagated through `map_keys(map_col)` and `map_values(map_col)`, producing paths like `[map_col, KEYS, NULL]` / `[map_col, VALUES, NULL]` (the new regression asserts `map_col.KEYS.NULL`). But the nullness of `map_keys(nullable_map)` or `map_values(nullable_map)` is determined by the parent map null map, not by the key/value child column. In BE, `MapFileColumnIterator::set_access_paths()` only switches to `NULL_MAP_ONLY` when the first sub-path component after the map name is `NULL`; with `KEYS.NULL`/`VALUES.NULL` it routes the path to the key/value iterator and still reads offsets plus child data (or relies on scalar fallback). For `map_keys(map_col) IS NULL` / `IS NOT NULL`, FE should emit/use the parent `[map_col, NULL]` path, or otherwise add BE handling that treats these function-level null checks as parent-map null-map-only reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4412140271
<details>
<summary>TPC-H: <b>Total hot run time: 29480 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f0cfe863011095f809f57d6db6e0a86c2de5ca1a, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 18173 3983 3890 3890
q2 q3 10732 870 611 611
q4 4660 456 346 346
q5 7451 1349 1161 1161
q6 192 172 142 142
q7 910 938 763 763
q8 9316 1419 1260 1260
q9 5624 5396 5339 5339
q10 6261 2078 1799 1799
q11 461 266 261 261
q12 630 406 291 291
q13 18065 3383 2722 2722
q14 300 286 263 263
q15 q16 865 847 794 794
q17 910 945 745 745
q18 6504 5665 5507 5507
q19 1167 1260 1051 1051
q20 502 412 265 265
q21 4627 2430 1935 1935
q22 481 423 335 335
Total cold run time: 97831 ms
Total hot run time: 29480 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4820 4769 4810 4769
q2 q3 4662 4800 4182 4182
q4 2106 2198 1396 1396
q5 4997 5005 5240 5005
q6 193 170 149 149
q7 2103 1805 1601 1601
q8 3362 3053 3087 3053
q9 8405 8475 8510 8475
q10 4496 4482 4289 4289
q11 613 410 386 386
q12 692 761 507 507
q13 3291 3633 2977 2977
q14 311 319 274 274
q15 q16 758 810 690 690
q17 1477 1285 1257 1257
q18 8029 7196 7106 7106
q19 1170 1139 1153 1139
q20 2248 2232 1951 1951
q21 6103 5424 4868 4868
q22 555 503 430 430
Total cold run time: 60391 ms
Total hot run time: 54504 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414072361
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4421062098
<details>
<summary>TPC-H: <b>Total hot run time: 29467 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 83d42d9aa645228fb4135a059ce26a489db011c4, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17617 3844 3780 3780
q2 q3 10699 862 597 597
q4 4678 455 339 339
q5 7447 1316 1133 1133
q6 184 165 138 138
q7 886 951 723 723
q8 9302 1364 1258 1258
q9 5550 5398 5301 5301
q10 6328 2079 1805 1805
q11 472 273 261 261
q12 689 410 281 281
q13 18190 3216 2733 2733
q14 292 278 263 263
q15 q16 893 870 790 790
q17 1019 947 777 777
q18 6344 5677 5589 5589
q19 1204 1272 1104 1104
q20 505 400 276 276
q21 4637 2336 1994 1994
q22 458 394 325 325
Total cold run time: 97394 ms
Total hot run time: 29467 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4737 4483 4522 4483
q2 q3 4630 4753 4182 4182
q4 2120 2166 1418 1418
q5 5176 4994 5273 4994
q6 190 167 133 133
q7 2049 1798 1662 1662
q8 3346 3072 3113 3072
q9 8643 8492 8391 8391
q10 4482 4457 4230 4230
q11 580 426 403 403
q12 716 772 534 534
q13 3226 3523 2976 2976
q14 298 299 274 274
q15 q16 743 780 684 684
q17 1362 1297 1247 1247
q18 7971 7076 7074 7074
q19 1158 1136 1155 1136
q20 2209 2232 1989 1989
q21 5992 5255 4750 4750
q22 523 476 404 404
Total cold run time: 60151 ms
Total hot run time: 54036 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4313939384
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4853573094
This PR implements the IS NULL optimization by removing the commented-out META-type `visitIsNull` and re-implementing it as an `ACCESS_NULL` path suffix (`[col, NULL]`). The removed version was:
// context.setType(ColumnAccessPathType.META);
Same "read-mode in the path string" direction as the OFFSET tier (#62205) — and this is the NULL half of the namespace collision I filed there: a struct field literally named `null` gets read as the NULL meta by BE's case-insensitive `StringCaseEqual`, so the struct enters `NULL_MAP_ONLY` and skips all sub-columns. Verified on a current build:
```sql
CREATE TABLE t(id INT, s STRUCT<`null`:INT, x:INT>)
DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES("replication_num"="1");
INSERT INTO t VALUES(1, named_struct('null',100,'x',200));
SELECT s.`null` FROM t; -- pruning ON -> NULL (wrong); OFF -> 100 (correct)
```
Two things specific to this PR:
1. The `stripNullSuffixPaths` comment here already documents the BE trap verbatim ("Struct/Array/Map iterators treat a leading NULL sub-path as NULL_MAP_ONLY and skip all children") — the behavior was known, but only the legitimate-meta path was guarded; the field-name entry was missed.
2. NULL is more dangerous than OFFSET: the struct reader actually acts on `NULL_MAP_ONLY`, while it has no `OFFSET_ONLY` branch — so an `offset` field survives by luck but a `null` field does not.
The functional IS NULL pruning itself tests correct (scalar / struct / variant / mixed / IS NOT NULL / WHERE pushdown all match the non-pruned results). The concern is the design direction: encoding read-mode as a path component reintroduces the namespace it then collides on. Keeping read-mode as the orthogonal `DATA`/`META` type dimension — the one this PR removed — avoids it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org