You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2026/04/10 02:39:27 UTC

[PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

englefly opened a new pull request, #62304:
URL: https://github.com/apache/doris/pull/62304

   ### What problem does this PR solve?
   Treat nullable fields as a combination of a nullable flag and data. When evaluating the `col IS NULL` predicate, use the NestedColumnPruning rule to prune the col field to col.NULL, thereby saving I/O on the data.
   
   Issue Number: close #xxx
   
   Related PR: #xxx
   
   Problem Summary:
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [ ] Regression test
       - [ ] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
   - Behavior changed:
       - [ ] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [ ] No.
       - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4317552504

   # FE Regression Coverage Report
   Increment line coverage `  88.79% (95/107)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_f8310449fd5a0bd83cf347a7e80a657fb1277ad2_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_f8310449fd5a0bd83cf347a7e80a657fb1277ad2_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4411864931

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414233406

   # FE UT Coverage Report
   Increment line coverage `  76.61% (131/171)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_8a43bbd701076d5edc3c81c31cb527861fe4040e/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_8a43bbd701076d5edc3c81c31cb527861fe4040e/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62304:
URL: https://github.com/apache/doris/pull/62304#discussion_r3097854572


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##########
@@ -286,6 +340,10 @@ private static Map<Integer, AccessPathInfo> pruneDataType(
                     });

Review Comment:
   `stripNullSuffixPaths()` breaks mixed projection/filter cases. BE readers use `all_access_paths` to decide which nested children are materialized, and `predicate_access_paths` is only forwarded after that child has already been selected. For example, `select struct_element(s, 'data') from tbl where struct_element(s, 'city') is null` now becomes `all=[s.data]` and `predicate=[s.city.NULL]`; `StructFileColumnIterator::set_access_paths()` will skip `city` entirely and the filter will run on default-filled data. Please keep the NULL path in `allAccessPaths` for mixed-access cases, or teach the BE readers to treat predicate-only subpaths as readable.



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java:
##########
@@ -72,7 +72,8 @@ public Map<Slot, List<CollectAccessPathResult>> collect(Plan root, StatementCont
     private boolean shouldCollectAccessPath(Slot slot) {
         return slot.getDataType() instanceof NestedColumnPrunable
                 || slot.getDataType().isVariantType()
-                || slot.getDataType().isStringLikeType();
+                || slot.getDataType().isStringLikeType()
+                || slot.nullable();

Review Comment:
   This broadens the optimization to every nullable scalar, but the BE scalar reader path still does not honor `[col, NULL]`. For OLAP scans we eventually land in `FileColumnIterator::set_access_paths()`, and that default implementation ignores `all_access_paths` completely, so `INT/DATE/... IS NULL` still reads full data pages. The new explain-only cases for `int_col.NULL` therefore look optimized even though no I/O is saved. Please either keep this scoped to readers with actual `NULL_MAP_ONLY` support, or add the missing BE scalar-reader handling in the same PR.



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##########
@@ -472,6 +556,17 @@ public boolean hasStringOffsetOnlyAccess() {
             return type.isStringLikeType() && isStringOffsetOnly && !accessAll;
         }
 
+        /** True when the column is accessed ONLY via IS NULL / IS NOT NULL,
+         *  meaning the BE only needs to read the null flag, not the actual data. */
+        public boolean hasNullCheckOnlyAccess() {
+            if (isRoot) {
+                DataTypeAccessTree child = children.values().iterator().next();
+                return child.isNullCheckOnly && !child.accessAll
+                        && !child.isStringOffsetOnly && !child.accessPartialChild;
+            }
+            return isNullCheckOnly && !accessAll && !isStringOffsetOnly && !accessPartialChild;
+        }
+
         /** pruneCastType */

Review Comment:
   For a top-level string predicate like `str_col IS NULL`, this never becomes `hasNullCheckOnlyAccess() == true`. By the time we hit this branch, `accessPartialChild` has already been set in the method prologue, so the root check at lines 561-567 still returns false. Because string slots take the dedicated branch earlier in `pruneDataType()` and `continue`, FE ends up attaching no access path at all, so the optimization does not trigger for top-level string columns. Could we avoid setting `accessPartialChild` for terminal `NULL` markers, or relax `hasNullCheckOnlyAccess()` for leaf nodes?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4317382703

   # FE UT Coverage Report
   Increment line coverage `  89.72% (96/107)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_f8310449fd5a0bd83cf347a7e80a657fb1277ad2/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_f8310449fd5a0bd83cf347a7e80a657fb1277ad2/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4417740756

   # BE Regression && UT Coverage Report
   Increment line coverage `100% (0/0)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.61% (27790/37755) |
   | Line Coverage     | 57.56% (300874/522736) |
   | Region Coverage   | 54.84% (251062/457768) |
   | Branch Coverage   | 56.32% (108425/192506) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4420520779

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4411792953

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4412162899

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 170878 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit f0cfe863011095f809f57d6db6e0a86c2de5ca1a, data reload: false
   
   query5	4340	656	517	517
   query6	324	215	199	199
   query7	4273	574	344	344
   query8	356	227	214	214
   query9	8813	3970	4004	3970
   query10	446	340	295	295
   query11	5790	2427	2199	2199
   query12	190	130	125	125
   query13	1302	593	438	438
   query14	5974	5391	5090	5090
   query14_1	4420	4385	4379	4379
   query15	212	203	188	188
   query16	1007	462	461	461
   query17	1159	800	635	635
   query18	2464	495	366	366
   query19	223	209	180	180
   query20	142	137	133	133
   query21	219	147	118	118
   query22	13576	13967	14593	13967
   query23	17268	16602	16180	16180
   query23_1	16341	16328	16327	16327
   query24	7471	1752	1395	1395
   query24_1	1341	1342	1344	1342
   query25	547	507	430	430
   query26	1317	318	167	167
   query27	2682	611	332	332
   query28	4386	1970	1941	1941
   query29	988	622	515	515
   query30	296	237	193	193
   query31	1091	1067	924	924
   query32	78	72	68	68
   query33	548	339	279	279
   query34	1167	1130	641	641
   query35	754	776	664	664
   query36	1324	1347	1239	1239
   query37	153	98	91	91
   query38	3202	3128	3051	3051
   query39	921	911	885	885
   query39_1	869	867	874	867
   query40	231	157	135	135
   query41	63	59	58	58
   query42	111	105	108	105
   query43	323	320	281	281
   query44	
   query45	208	208	195	195
   query46	1096	1163	708	708
   query47	2327	2301	2221	2221
   query48	400	409	295	295
   query49	628	516	432	432
   query50	707	290	212	212
   query51	4319	4248	4212	4212
   query52	104	102	92	92
   query53	267	272	201	201
   query54	312	271	250	250
   query55	93	92	87	87
   query56	303	302	304	302
   query57	1407	1408	1302	1302
   query58	285	268	282	268
   query59	1541	1630	1442	1442
   query60	341	338	320	320
   query61	160	158	156	156
   query62	654	624	570	570
   query63	241	198	201	198
   query64	2436	817	672	672
   query65	
   query66	1765	504	388	388
   query67	30064	29973	29781	29781
   query68	
   query69	459	336	296	296
   query70	1016	982	903	903
   query71	308	273	274	273
   query72	2960	2737	2461	2461
   query73	855	737	402	402
   query74	5086	4920	4726	4726
   query75	2770	2656	2339	2339
   query76	2294	1128	781	781
   query77	405	428	342	342
   query78	12925	12996	12255	12255
   query79	1493	991	742	742
   query80	1371	598	494	494
   query81	504	278	239	239
   query82	1315	158	122	122
   query83	356	274	245	245
   query84	271	138	108	108
   query85	898	524	445	445
   query86	456	330	327	327
   query87	3434	3356	3214	3214
   query88	3490	2669	2642	2642
   query89	439	383	336	336
   query90	1956	176	181	176
   query91	180	172	140	140
   query92	79	75	71	71
   query93	960	961	561	561
   query94	716	362	307	307
   query95	663	383	366	366
   query96	1022	776	342	342
   query97	2712	2669	2576	2576
   query98	247	235	232	232
   query99	1140	1107	987	987
   Total cold run time: 254316 ms
   Total hot run time: 170878 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4412008514

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29711 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit f0cfe863011095f809f57d6db6e0a86c2de5ca1a, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17724	3930	4084	3930
   q2	q3	10727	863	605	605
   q4	4656	460	354	354
   q5	7450	1339	1151	1151
   q6	196	165	145	145
   q7	933	930	748	748
   q8	9662	1378	1315	1315
   q9	6450	5369	5349	5349
   q10	6335	2069	1827	1827
   q11	474	264	255	255
   q12	682	434	294	294
   q13	18165	3286	2741	2741
   q14	291	283	263	263
   q15	q16	906	880	789	789
   q17	1060	1087	742	742
   q18	6503	5701	5643	5643
   q19	1712	1271	988	988
   q20	511	393	262	262
   q21	4952	2352	1968	1968
   q22	477	389	342	342
   Total cold run time: 99866 ms
   Total hot run time: 29711 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4692	4540	4603	4540
   q2	q3	4645	4793	4235	4235
   q4	2197	2173	1406	1406
   q5	5017	4992	5281	4992
   q6	200	173	143	143
   q7	2001	1808	1603	1603
   q8	3368	3099	3101	3099
   q9	8510	8688	8507	8507
   q10	4498	4523	4263	4263
   q11	598	416	391	391
   q12	726	745	562	562
   q13	3246	3586	3000	3000
   q14	307	307	276	276
   q15	q16	768	781	706	706
   q17	1387	1341	1306	1306
   q18	7947	7156	7121	7121
   q19	1177	1151	1168	1151
   q20	2231	2228	1988	1988
   q21	6111	5293	4875	4875
   q22	528	483	408	408
   Total cold run time: 60154 ms
   Total hot run time: 54572 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4395480024

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4402499224

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #62304:
URL: https://github.com/apache/doris/pull/62304#discussion_r3214105097


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java:
##########
@@ -506,14 +546,35 @@ public Void visitArraySortBy(ArraySortBy arraySortBy, CollectorContext context)
         return visit(arraySortBy, context);
     }
 
-    // @Override
-    // public Void visitIsNull(IsNull isNull, CollectorContext context) {
-    //     if (context.accessPathBuilder.isEmpty()) {
-    //         context.setType(ColumnAccessPathType.META);
-    //         return continueCollectAccessPath(isNull.child(), context);
-    //     }
-    //     return visit(isNull, context);
-    // }
+    @Override
+    public Void visitIsNull(IsNull isNull, CollectorContext context) {
+        Expression arg = isNull.child();
+        // Skip variant sub-column paths (v['k'] IS NULL): the sub-column path is already baked
+        // into the SlotReference, so null-only access doesn't apply the same way.
+        if (arg instanceof SlotReference && ((SlotReference) arg).hasSubColPath()) {
+            return visit(isNull, context);
+        }
+        // Optimize IS NULL on nullable expressions: create a context with NULL suffix to indicate
+        // only the null flag is needed. Works for top-level columns (col IS NULL → [col, NULL])
+        // and nested access (struct_element(s, 'city') IS NULL → [s, city, NULL]).
+        // For unrecognized expressions, the default visitor resets context, safely discarding NULL.
+        if (arg.nullable() && context.accessPathBuilder.isEmpty()) {
+            CollectorContext nullContext =
+                    new CollectorContext(context.statementContext, context.bottomFilter);
+            nullContext.accessPathBuilder.addSuffix(AccessPathInfo.ACCESS_NULL);
+            return continueCollectAccessPath(arg, nullContext);

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414072128

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4428630883

   # BE Regression && UT Coverage Report
   Increment line coverage `100% (0/0)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.70% (27800/37718) |
   | Line Coverage     | 57.55% (300824/522712) |
   | Region Coverage   | 54.72% (250766/458245) |
   | Branch Coverage   | 56.26% (108508/192861) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4427157915

   run beut


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4265080344

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4411793167

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4411865329

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414121517

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29667 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 8a43bbd701076d5edc3c81c31cb527861fe4040e, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17734	3882	3857	3857
   q2	q3	10716	900	620	620
   q4	4671	464	362	362
   q5	7501	1368	1167	1167
   q6	204	168	143	143
   q7	953	960	789	789
   q8	9519	1431	1340	1340
   q9	6089	5426	5369	5369
   q10	6297	2087	1822	1822
   q11	481	275	273	273
   q12	699	431	311	311
   q13	18164	3326	2744	2744
   q14	295	292	266	266
   q15	q16	924	860	788	788
   q17	1029	1081	689	689
   q18	6471	5707	5575	5575
   q19	1178	1242	1114	1114
   q20	501	403	263	263
   q21	4680	2289	1869	1869
   q22	419	354	306	306
   Total cold run time: 98525 ms
   Total hot run time: 29667 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4191	4125	4230	4125
   q2	q3	4628	4776	4157	4157
   q4	2090	2171	1412	1412
   q5	4968	5039	5272	5039
   q6	185	162	136	136
   q7	2028	1792	1742	1742
   q8	3484	3213	3172	3172
   q9	8485	8509	8316	8316
   q10	4484	4503	4261	4261
   q11	584	422	418	418
   q12	717	772	543	543
   q13	3243	3590	2941	2941
   q14	307	306	266	266
   q15	q16	767	780	678	678
   q17	1347	1322	1290	1290
   q18	8041	7164	7107	7107
   q19	1164	1155	1183	1155
   q20	2268	2232	1959	1959
   q21	6100	5408	4905	4905
   q22	553	540	423	423
   Total cold run time: 59634 ms
   Total hot run time: 54045 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414138300

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 171552 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 8a43bbd701076d5edc3c81c31cb527861fe4040e, data reload: false
   
   query5	4327	668	517	517
   query6	348	248	221	221
   query7	4239	583	312	312
   query8	336	244	238	238
   query9	8829	4122	4041	4041
   query10	501	355	306	306
   query11	6085	2393	2211	2211
   query12	194	133	132	132
   query13	1288	614	464	464
   query14	6885	5419	5104	5104
   query14_1	4430	4437	4370	4370
   query15	213	210	189	189
   query16	968	461	445	445
   query17	1388	782	658	658
   query18	2742	511	366	366
   query19	322	217	170	170
   query20	141	136	130	130
   query21	217	143	119	119
   query22	13621	13586	13340	13340
   query23	17198	16365	16591	16365
   query23_1	16281	16310	16411	16310
   query24	7963	1855	1392	1392
   query24_1	1465	1401	1421	1401
   query25	589	509	461	461
   query26	1323	348	190	190
   query27	2851	639	362	362
   query28	4360	2012	1980	1980
   query29	1017	653	531	531
   query30	301	233	197	197
   query31	1125	1079	941	941
   query32	81	75	73	73
   query33	524	341	287	287
   query34	1122	1137	643	643
   query35	768	791	671	671
   query36	1312	1274	1135	1135
   query37	149	100	89	89
   query38	3227	3173	3071	3071
   query39	936	935	929	929
   query39_1	885	867	868	867
   query40	234	153	137	137
   query41	63	62	62	62
   query42	112	106	110	106
   query43	333	332	288	288
   query44	
   query45	210	207	199	199
   query46	1129	1212	731	731
   query47	2294	2271	2208	2208
   query48	417	429	277	277
   query49	631	531	459	459
   query50	756	307	236	236
   query51	4416	4337	4349	4337
   query52	110	109	99	99
   query53	263	286	211	211
   query54	312	277	254	254
   query55	96	94	84	84
   query56	309	300	322	300
   query57	1455	1408	1268	1268
   query58	290	273	266	266
   query59	1606	1651	1463	1463
   query60	343	342	324	324
   query61	151	157	155	155
   query62	664	617	569	569
   query63	246	205	205	205
   query64	2318	835	698	698
   query65	
   query66	1667	508	407	407
   query67	30274	30090	29911	29911
   query68	
   query69	469	333	304	304
   query70	1006	1027	966	966
   query71	310	282	275	275
   query72	2933	2679	2399	2399
   query73	836	765	423	423
   query74	5072	4914	4758	4758
   query75	2758	2673	2344	2344
   query76	2289	1155	818	818
   query77	417	433	340	340
   query78	12859	12984	12502	12502
   query79	1497	1001	757	757
   query80	785	600	499	499
   query81	469	285	239	239
   query82	1255	157	121	121
   query83	363	283	256	256
   query84	264	144	109	109
   query85	885	483	449	449
   query86	414	352	313	313
   query87	3443	3348	3221	3221
   query88	3593	2661	2649	2649
   query89	440	385	340	340
   query90	1796	179	188	179
   query91	178	169	136	136
   query92	79	79	74	74
   query93	994	976	566	566
   query94	596	333	295	295
   query95	665	448	347	347
   query96	1008	763	351	351
   query97	2687	2683	2612	2612
   query98	239	229	227	227
   query99	1095	1115	981	981
   Total cold run time: 255735 ms
   Total hot run time: 171552 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4417029669

   # BE UT Coverage Report
   Increment line coverage ` ` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 53.57% (20652/38551) |
   | Line Coverage     | 37.19% (194906/524109) |
   | Region Coverage   | 33.50% (151884/453348) |
   | Branch Coverage   | 34.55% (66264/191778) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4418099162

   run external


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4317437593

   OpenCode automated review failed and did not complete.
   
   Error: Review step was failure (possibly timeout or cancelled)
   Workflow run: https://github.com/apache/doris/actions/runs/24916998608
   
   Please inspect the workflow logs and rerun the review after the underlying issue is resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4314912686

   # FE Regression Coverage Report
   Increment line coverage `  88.79% (95/107)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_66069bc6fa759d2a07384e6438aa9a765d229066_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_66069bc6fa759d2a07384e6438aa9a765d229066_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #62304:
URL: https://github.com/apache/doris/pull/62304


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4265079868

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62304:
URL: https://github.com/apache/doris/pull/62304#discussion_r3212691845


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##########
@@ -246,7 +301,22 @@ private static Map<Integer, AccessPathInfo> pruneDataType(
                 continue;
             }
 
+            // Null-check-only access (e.g. col IS NULL / col IS NOT NULL): type stays unchanged,
+            // but we must send the [col, NULL] access path to BE so it only reads the null flag.
+            if (accessTree.hasNullCheckOnlyAccess()) {
+                if (skipDataSkippingOnlyAccessPath) {

Review Comment:
   This MV-fragment skip only catches top-level `col IS NULL`/`IS NOT NULL` cases. For a nested predicate like `struct_element(s, 'city') is not null`, `hasNullCheckOnlyAccess()` is false because the root `s` has `accessPartialChild`, so the code falls through and still emits `[s.city.NULL]` plus a pruned `struct<city:...>` type even when `skipDataSkippingOnlyAccessPath` is true. That can reproduce the same MV temporary-fragment bug this flag is trying to avoid: the temporary rewritten fragment only needs the nested null flag for filtering, but the final MV rewritten plan may still need the full complex slot or another child. This is distinct from the existing parent-NULL/allAccessPath threads; it is the MV-rewrite skip path not covering nested NULL-only access. Please either suppress nested data-skipping-only access paths under this flag as well, or add a nested MV rewrite regression that proves the final plan cannot reuse a pruned complex slot.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414346844

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4416929210

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 170101 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016, data reload: false
   
   query5	4338	653	524	524
   query6	327	216	200	200
   query7	4275	558	323	323
   query8	326	231	221	221
   query9	8826	4183	4126	4126
   query10	467	356	305	305
   query11	5818	2411	2255	2255
   query12	185	131	134	131
   query13	1278	622	446	446
   query14	6269	5381	5123	5123
   query14_1	4417	4397	4349	4349
   query15	215	204	184	184
   query16	1032	472	444	444
   query17	1150	791	617	617
   query18	2743	478	340	340
   query19	213	198	157	157
   query20	138	131	131	131
   query21	212	135	113	113
   query22	13649	13646	13371	13371
   query23	17165	16357	15907	15907
   query23_1	16137	16164	16210	16164
   query24	7393	1738	1348	1348
   query24_1	1357	1373	1343	1343
   query25	551	496	442	442
   query26	1311	304	167	167
   query27	2769	583	333	333
   query28	4409	2018	1967	1967
   query29	1002	639	512	512
   query30	311	234	198	198
   query31	1127	1064	928	928
   query32	86	73	75	73
   query33	530	349	286	286
   query34	1164	1106	635	635
   query35	768	789	666	666
   query36	1330	1307	1211	1211
   query37	158	108	92	92
   query38	3198	3157	3051	3051
   query39	974	971	902	902
   query39_1	904	912	934	912
   query40	238	156	139	139
   query41	68	64	61	61
   query42	115	108	111	108
   query43	328	341	303	303
   query44	
   query45	206	203	195	195
   query46	1074	1217	789	789
   query47	2332	2355	2151	2151
   query48	413	439	306	306
   query49	676	547	449	449
   query50	719	291	223	223
   query51	4287	4303	4306	4303
   query52	107	108	95	95
   query53	262	291	214	214
   query54	333	298	275	275
   query55	103	90	86	86
   query56	318	319	322	319
   query57	1402	1397	1305	1305
   query58	305	281	265	265
   query59	1577	1622	1430	1430
   query60	359	351	332	332
   query61	180	175	179	175
   query62	687	629	559	559
   query63	249	200	210	200
   query64	2425	860	735	735
   query65	
   query66	1699	538	417	417
   query67	29965	29300	29832	29300
   query68	
   query69	468	330	294	294
   query70	960	1010	1002	1002
   query71	310	276	270	270
   query72	2940	2758	2470	2470
   query73	862	779	445	445
   query74	5087	4884	4725	4725
   query75	2798	2686	2344	2344
   query76	2280	1137	742	742
   query77	420	417	352	352
   query78	12950	12996	12386	12386
   query79	1503	987	773	773
   query80	1410	594	493	493
   query81	537	281	240	240
   query82	1121	157	123	123
   query83	327	292	254	254
   query84	265	148	116	116
   query85	931	518	454	454
   query86	463	322	330	322
   query87	3442	3367	3230	3230
   query88	3574	2767	2704	2704
   query89	458	377	338	338
   query90	1957	179	183	179
   query91	177	174	141	141
   query92	82	78	74	74
   query93	1062	966	564	564
   query94	717	343	295	295
   query95	687	362	351	351
   query96	997	781	317	317
   query97	2719	2671	2566	2566
   query98	243	233	244	233
   query99	1150	1120	1007	1007
   Total cold run time: 255066 ms
   Total hot run time: 170101 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414410554

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 171966 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 8a43bbd701076d5edc3c81c31cb527861fe4040e, data reload: false
   
   query5	4339	654	527	527
   query6	335	237	215	215
   query7	4226	552	315	315
   query8	341	246	231	231
   query9	8859	4110	4091	4091
   query10	447	347	309	309
   query11	5763	2390	2211	2211
   query12	183	135	131	131
   query13	1281	633	440	440
   query14	6744	5334	5080	5080
   query14_1	4342	4378	4344	4344
   query15	217	205	189	189
   query16	1012	457	453	453
   query17	1157	774	649	649
   query18	2777	500	372	372
   query19	233	212	172	172
   query20	140	135	133	133
   query21	216	138	120	120
   query22	13569	13592	13357	13357
   query23	17179	16434	16479	16434
   query23_1	16409	16279	16320	16279
   query24	7870	1819	1416	1416
   query24_1	1416	1447	1414	1414
   query25	614	570	497	497
   query26	1411	373	179	179
   query27	3154	618	345	345
   query28	4451	2030	1960	1960
   query29	1057	657	527	527
   query30	317	248	191	191
   query31	1111	1071	932	932
   query32	84	67	71	67
   query33	546	344	298	298
   query34	1179	1164	609	609
   query35	766	784	680	680
   query36	1361	1322	1137	1137
   query37	147	98	84	84
   query38	3158	3109	3106	3106
   query39	917	915	891	891
   query39_1	859	869	875	869
   query40	234	159	131	131
   query41	62	64	57	57
   query42	109	110	107	107
   query43	322	323	277	277
   query44	
   query45	205	196	188	188
   query46	1083	1198	728	728
   query47	2359	2297	2284	2284
   query48	415	438	286	286
   query49	640	531	437	437
   query50	730	312	223	223
   query51	4317	4306	4309	4306
   query52	108	104	92	92
   query53	253	297	211	211
   query54	312	268	256	256
   query55	95	100	83	83
   query56	311	338	323	323
   query57	1486	1445	1353	1353
   query58	301	271	274	271
   query59	1633	1710	1458	1458
   query60	360	349	335	335
   query61	147	154	161	154
   query62	681	633	562	562
   query63	247	206	208	206
   query64	2348	829	677	677
   query65	
   query66	1694	529	403	403
   query67	30038	29964	29869	29869
   query68	
   query69	455	354	302	302
   query70	990	978	1000	978
   query71	306	283	276	276
   query72	2944	2665	2450	2450
   query73	823	759	445	445
   query74	5088	4873	4791	4791
   query75	2811	2650	2320	2320
   query76	2308	1138	763	763
   query77	414	406	348	348
   query78	12918	13004	12374	12374
   query79	1518	1018	717	717
   query80	1384	603	488	488
   query81	521	292	240	240
   query82	1075	165	119	119
   query83	353	278	256	256
   query84	274	139	118	118
   query85	961	516	447	447
   query86	464	321	306	306
   query87	3430	3350	3244	3244
   query88	3585	2695	2683	2683
   query89	438	380	339	339
   query90	1968	179	175	175
   query91	191	164	143	143
   query92	76	76	68	68
   query93	1049	986	592	592
   query94	724	305	304	304
   query95	675	369	344	344
   query96	1047	812	351	351
   query97	2729	2720	2620	2620
   query98	242	235	236	235
   query99	1123	1098	984	984
   Total cold run time: 256338 ms
   Total hot run time: 171966 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4417370563

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4421153907

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 170638 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 83d42d9aa645228fb4135a059ce26a489db011c4, data reload: false
   
   query5	4307	652	524	524
   query6	334	229	193	193
   query7	4214	560	305	305
   query8	337	230	214	214
   query9	8815	3976	3964	3964
   query10	430	332	304	304
   query11	5824	2395	2221	2221
   query12	194	138	133	133
   query13	1301	613	450	450
   query14	5950	5367	5090	5090
   query14_1	4364	4395	4370	4370
   query15	215	202	182	182
   query16	1011	448	432	432
   query17	1164	790	634	634
   query18	2507	496	378	378
   query19	215	212	170	170
   query20	142	139	130	130
   query21	230	139	120	120
   query22	13638	13692	13377	13377
   query23	17316	16355	16000	16000
   query23_1	16250	16197	16177	16177
   query24	7463	1750	1350	1350
   query24_1	1367	1367	1341	1341
   query25	594	519	480	480
   query26	1309	293	172	172
   query27	2715	620	345	345
   query28	4426	1958	1935	1935
   query29	1014	640	515	515
   query30	309	240	199	199
   query31	1107	1064	942	942
   query32	85	78	73	73
   query33	535	354	301	301
   query34	1163	1102	656	656
   query35	767	790	685	685
   query36	1301	1333	1137	1137
   query37	157	97	86	86
   query38	3228	3077	3056	3056
   query39	926	936	891	891
   query39_1	888	874	873	873
   query40	241	160	135	135
   query41	66	62	63	62
   query42	115	111	109	109
   query43	326	327	290	290
   query44	
   query45	213	199	192	192
   query46	1085	1189	706	706
   query47	2376	2403	2278	2278
   query48	406	420	291	291
   query49	663	549	469	469
   query50	715	303	232	232
   query51	4399	4378	4245	4245
   query52	113	109	95	95
   query53	251	280	205	205
   query54	311	272	248	248
   query55	92	88	83	83
   query56	305	319	328	319
   query57	1404	1406	1326	1326
   query58	311	283	267	267
   query59	1542	1605	1390	1390
   query60	339	338	327	327
   query61	162	156	158	156
   query62	668	616	561	561
   query63	240	205	217	205
   query64	2456	837	685	685
   query65	
   query66	1764	508	384	384
   query67	30039	29366	29794	29366
   query68	
   query69	476	336	301	301
   query70	998	1051	995	995
   query71	305	279	269	269
   query72	2917	2698	2404	2404
   query73	824	771	411	411
   query74	5035	4903	4743	4743
   query75	2786	2676	2341	2341
   query76	2280	1123	774	774
   query77	425	428	363	363
   query78	12951	12964	12337	12337
   query79	1493	984	749	749
   query80	966	620	511	511
   query81	503	316	244	244
   query82	1332	162	127	127
   query83	362	275	244	244
   query84	255	145	115	115
   query85	908	508	435	435
   query86	422	359	330	330
   query87	3414	3344	3219	3219
   query88	3521	2666	2645	2645
   query89	450	387	347	347
   query90	1813	177	185	177
   query91	173	170	137	137
   query92	82	78	74	74
   query93	955	948	566	566
   query94	617	350	298	298
   query95	659	376	427	376
   query96	1011	756	360	360
   query97	2718	2719	2574	2574
   query98	248	236	233	233
   query99	1114	1122	993	993
   Total cold run time: 253674 ms
   Total hot run time: 170638 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4425368319

   # FE Regression Coverage Report
   Increment line coverage `  79.79% (150/188)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4384723759

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4393399696

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4340061645

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4415595086

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29660 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 54a413523493235f44674286e28354973dfab413, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17596	3973	3984	3973
   q2	q3	10716	868	604	604
   q4	4660	464	353	353
   q5	7441	1339	1171	1171
   q6	199	181	151	151
   q7	929	982	744	744
   q8	9331	1464	1237	1237
   q9	5668	5386	5303	5303
   q10	6245	2062	1799	1799
   q11	468	268	255	255
   q12	637	439	305	305
   q13	18118	3333	2702	2702
   q14	294	291	271	271
   q15	q16	885	864	796	796
   q17	1001	1062	735	735
   q18	6405	5738	5603	5603
   q19	1253	1226	1110	1110
   q20	528	395	256	256
   q21	4553	2465	1963	1963
   q22	492	381	329	329
   Total cold run time: 97419 ms
   Total hot run time: 29660 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	5066	4786	5049	4786
   q2	q3	4658	4805	4241	4241
   q4	2312	2240	1451	1451
   q5	4999	4997	5199	4997
   q6	203	184	141	141
   q7	2104	1879	1622	1622
   q8	3323	3112	3123	3112
   q9	8452	8435	8494	8435
   q10	4568	4549	4291	4291
   q11	613	429	429	429
   q12	704	757	541	541
   q13	3222	3608	2898	2898
   q14	300	299	298	298
   q15	q16	928	774	723	723
   q17	1373	1367	1293	1293
   q18	8049	7186	7210	7186
   q19	1164	1150	1191	1150
   q20	2287	2270	1963	1963
   q21	6358	5628	5013	5013
   q22	582	504	437	437
   Total cold run time: 61265 ms
   Total hot run time: 55007 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4415749628

   # FE UT Coverage Report
   Increment line coverage `  76.61% (131/171)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_54a413523493235f44674286e28354973dfab413/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_54a413523493235f44674286e28354973dfab413/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62304:
URL: https://github.com/apache/doris/pull/62304#discussion_r3205665945


##########
task.md:
##########
@@ -0,0 +1,15 @@
+当前分支实现了 is-null 谓词的优化.

Review Comment:
   This looks like a local scratch note describing the debugging task and stack trace, not a Doris source/test/doc change. Please remove it from the PR before merging.



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##########
@@ -480,6 +541,60 @@ private List<List<String>> getSupplementalPaths() {
         }

Review Comment:
   This parent-`NULL` path cannot be kept in `allAccessPaths` together with child data paths with the current BE contract. For `select struct_element(struct_col, 'city') from ncp_tbl where struct_col is null`, FE now sends `all=[struct_col.NULL, struct_col.city]`. In BE, `StructFileColumnIterator::set_access_paths()` strips the root name, sees `NULL` in `sub_all_access_paths`, `_check_and_set_meta_read_mode()` switches the whole struct iterator to `NULL_MAP_ONLY`, and lines 1562-1568 skip all sub-column iterators before `city` is routed. The projected `city` value is then default-filled instead of read. This is distinct from the existing predicate-only child-path thread: here the child path is present in `allAccessPaths`, but the parent `NULL` meta mode takes precedence and suppresses it. Please either avoid emitting parent `.NULL` in `allAccessPaths` when any real child path is also required, or update the BE iterator logic so `NULL_MAP_ONLY` only applies when the null-map path is t
 he only required access.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4416809255

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4417046766

   # FE UT Coverage Report
   Increment line coverage `  76.61% (131/171)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4416896005

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29459 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17809	3847	3910	3847
   q2	q3	10725	858	591	591
   q4	4659	460	336	336
   q5	7447	1323	1146	1146
   q6	193	170	139	139
   q7	890	948	754	754
   q8	9304	1409	1308	1308
   q9	5661	5349	5313	5313
   q10	6318	2086	1810	1810
   q11	476	262	258	258
   q12	661	418	290	290
   q13	18188	3253	2713	2713
   q14	285	282	264	264
   q15	q16	905	880	785	785
   q17	986	1099	778	778
   q18	6443	5685	5581	5581
   q19	1295	1218	1053	1053
   q20	513	396	252	252
   q21	4826	2370	1913	1913
   q22	475	382	328	328
   Total cold run time: 98059 ms
   Total hot run time: 29459 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4657	4540	4602	4540
   q2	q3	4637	4776	4180	4180
   q4	2113	2195	1399	1399
   q5	4967	5048	5290	5048
   q6	195	175	140	140
   q7	2075	1794	1629	1629
   q8	3339	3132	3118	3118
   q9	8504	8502	8439	8439
   q10	4481	4466	4227	4227
   q11	618	427	401	401
   q12	709	773	530	530
   q13	3211	3653	2972	2972
   q14	311	318	276	276
   q15	q16	778	769	693	693
   q17	1350	1357	1299	1299
   q18	7969	7205	7074	7074
   q19	1150	1169	1142	1142
   q20	2214	2241	1977	1977
   q21	6198	5458	4822	4822
   q22	535	480	402	402
   Total cold run time: 60011 ms
   Total hot run time: 54308 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4412031802

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 172661 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit f0cfe863011095f809f57d6db6e0a86c2de5ca1a, data reload: false
   
   query5	4310	687	526	526
   query6	332	246	225	225
   query7	4245	579	328	328
   query8	334	240	222	222
   query9	8812	4140	4080	4080
   query10	468	350	313	313
   query11	5908	2607	2258	2258
   query12	190	134	133	133
   query13	1299	632	434	434
   query14	6862	5398	5111	5111
   query14_1	4383	4384	4332	4332
   query15	210	207	182	182
   query16	1032	464	433	433
   query17	1194	785	623	623
   query18	2735	488	355	355
   query19	219	200	173	173
   query20	149	134	133	133
   query21	217	138	120	120
   query22	13618	13607	13377	13377
   query23	17249	16547	16038	16038
   query23_1	16187	16074	16145	16074
   query24	7453	1764	1357	1357
   query24_1	1387	1345	1403	1345
   query25	553	521	482	482
   query26	1319	322	179	179
   query27	2709	598	351	351
   query28	4408	2014	1989	1989
   query29	1022	695	561	561
   query30	310	247	205	205
   query31	1133	1068	958	958
   query32	92	81	77	77
   query33	547	372	318	318
   query34	1169	1142	673	673
   query35	771	798	695	695
   query36	1292	1337	1101	1101
   query37	162	112	99	99
   query38	3255	3181	3087	3087
   query39	943	967	936	936
   query39_1	906	919	912	912
   query40	248	164	157	157
   query41	76	70	68	68
   query42	123	123	127	123
   query43	350	348	298	298
   query44	
   query45	222	209	197	197
   query46	1170	1286	758	758
   query47	2281	2298	2149	2149
   query48	434	396	316	316
   query49	650	558	445	445
   query50	737	294	229	229
   query51	4314	4264	4208	4208
   query52	110	112	103	103
   query53	265	297	209	209
   query54	349	289	274	274
   query55	98	95	89	89
   query56	322	337	328	328
   query57	1412	1385	1337	1337
   query58	315	287	285	285
   query59	1592	1643	1407	1407
   query60	366	366	343	343
   query61	189	186	185	185
   query62	678	629	569	569
   query63	254	214	210	210
   query64	2414	829	680	680
   query65	
   query66	1699	513	408	408
   query67	30426	30031	30033	30031
   query68	
   query69	480	344	307	307
   query70	1063	1044	1030	1030
   query71	324	283	272	272
   query72	3008	2773	2451	2451
   query73	855	787	420	420
   query74	5134	4918	4796	4796
   query75	2801	2671	2347	2347
   query76	2308	1141	799	799
   query77	427	433	352	352
   query78	12986	13077	12422	12422
   query79	1514	999	764	764
   query80	843	601	489	489
   query81	478	281	244	244
   query82	1300	171	127	127
   query83	387	284	262	262
   query84	261	148	112	112
   query85	934	527	453	453
   query86	434	331	330	330
   query87	3443	3363	3230	3230
   query88	3606	2741	2675	2675
   query89	449	389	350	350
   query90	1781	186	182	182
   query91	180	172	145	145
   query92	82	76	75	75
   query93	971	973	593	593
   query94	614	352	313	313
   query95	672	469	362	362
   query96	1079	762	338	338
   query97	2665	2746	2598	2598
   query98	239	231	236	231
   query99	1118	1088	969	969
   Total cold run time: 255902 ms
   Total hot run time: 172661 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4426291155

   run beut


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4386088903

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4396171829

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4221088978

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4402499671

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4418786719

   # FE Regression Coverage Report
   Increment line coverage `  37.88% (136/359)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4422340660

   # FE UT Coverage Report
   Increment line coverage `  76.60% (144/188)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_83d42d9aa645228fb4135a059ce26a489db011c4/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414393146

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29345 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 8a43bbd701076d5edc3c81c31cb527861fe4040e, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17809	3876	3847	3847
   q2	q3	10739	870	601	601
   q4	4656	449	358	358
   q5	7465	1331	1133	1133
   q6	190	168	137	137
   q7	943	933	770	770
   q8	9297	1359	1209	1209
   q9	5661	5408	5290	5290
   q10	6298	2069	1802	1802
   q11	489	273	262	262
   q12	691	414	303	303
   q13	18174	3339	2713	2713
   q14	299	281	260	260
   q15	q16	901	863	797	797
   q17	1011	1085	712	712
   q18	6441	5688	5595	5595
   q19	1309	1289	1114	1114
   q20	507	391	263	263
   q21	4548	2450	1864	1864
   q22	426	353	315	315
   Total cold run time: 97854 ms
   Total hot run time: 29345 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4181	4096	4067	4067
   q2	q3	4614	4740	4200	4200
   q4	2104	2171	1390	1390
   q5	4968	4956	5246	4956
   q6	185	164	131	131
   q7	2038	1776	1909	1776
   q8	3437	3156	3152	3152
   q9	8523	8414	8495	8414
   q10	4454	4506	4262	4262
   q11	590	436	399	399
   q12	733	744	534	534
   q13	3153	3575	2954	2954
   q14	294	316	269	269
   q15	q16	761	783	710	710
   q17	1352	1312	1251	1251
   q18	7974	7032	7189	7032
   q19	1184	1154	1197	1154
   q20	2268	2288	1963	1963
   q21	6117	5449	4934	4934
   q22	564	511	426	426
   Total cold run time: 59494 ms
   Total hot run time: 53974 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4415559999

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4418712151

   # BE Regression && UT Coverage Report
   Increment line coverage `100% (0/0)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_b5fc0b568953dfe130f8b77cdd3f9cadcdc3f016_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.62% (27794/37755) |
   | Line Coverage     | 57.57% (300931/522736) |
   | Region Coverage   | 54.82% (250955/457768) |
   | Branch Coverage   | 56.32% (108414/192506) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4313938810

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4314707800

   OpenCode automated review failed and did not complete.
   
   Error: Review step was failure (possibly timeout or cancelled)
   Workflow run: https://github.com/apache/doris/actions/runs/24894832326
   
   Please inspect the workflow logs and rerun the review after the underlying issue is resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4221786883

   # FE UT Coverage Report
   Increment line coverage `  88.89% (72/81)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_5f535837cb9a5835a1b4e7bc310e3e91c260a3a6/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_5f535837cb9a5835a1b4e7bc310e3e91c260a3a6/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4219667555

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4265546186

   # FE UT Coverage Report
   Increment line coverage `  80.58% (83/103)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_eee072014d893e3257b248f16a29d6dba02a4429/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_eee072014d893e3257b248f16a29d6dba02a4429/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4393406856

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4411726102

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4428591768

   # BE UT Coverage Report
   Increment line coverage ` ` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/83d42d9aa645228fb4135a059ce26a489db011c4_83d42d9aa645228fb4135a059ce26a489db011c4/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/83d42d9aa645228fb4135a059ce26a489db011c4_83d42d9aa645228fb4135a059ce26a489db011c4/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 53.57% (20634/38519) |
   | Line Coverage     | 37.19% (194919/524097) |
   | Region Coverage   | 33.57% (152342/453769) |
   | Branch Coverage   | 34.58% (66432/192121) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4415617975

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 170728 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 54a413523493235f44674286e28354973dfab413, data reload: false
   
   query5	4329	673	538	538
   query6	353	230	214	214
   query7	4227	576	327	327
   query8	333	240	224	224
   query9	8848	4044	4002	4002
   query10	472	347	308	308
   query11	5779	2462	2214	2214
   query12	185	132	128	128
   query13	1271	647	455	455
   query14	6095	5366	5050	5050
   query14_1	4372	4402	4340	4340
   query15	219	211	196	196
   query16	983	493	375	375
   query17	1144	786	647	647
   query18	2513	523	380	380
   query19	226	208	178	178
   query20	144	136	134	134
   query21	220	138	119	119
   query22	13595	13853	14597	13853
   query23	17418	16528	16071	16071
   query23_1	16250	16307	16165	16165
   query24	7405	1800	1325	1325
   query24_1	1336	1342	1347	1342
   query25	570	489	425	425
   query26	1324	323	170	170
   query27	2731	610	353	353
   query28	4377	1940	1960	1940
   query29	987	626	513	513
   query30	304	226	191	191
   query31	1099	1063	931	931
   query32	83	70	71	70
   query33	540	345	281	281
   query34	1183	1152	639	639
   query35	783	796	668	668
   query36	1318	1329	1116	1116
   query37	151	110	84	84
   query38	3215	3137	3055	3055
   query39	931	923	879	879
   query39_1	863	883	868	868
   query40	233	152	137	137
   query41	62	60	58	58
   query42	109	107	106	106
   query43	335	328	284	284
   query44	
   query45	213	199	193	193
   query46	1092	1254	727	727
   query47	2245	2255	2187	2187
   query48	396	424	294	294
   query49	655	511	427	427
   query50	703	288	227	227
   query51	4331	4335	4163	4163
   query52	108	108	95	95
   query53	253	281	206	206
   query54	310	272	251	251
   query55	97	88	84	84
   query56	305	321	304	304
   query57	1416	1405	1318	1318
   query58	287	274	274	274
   query59	1564	1617	1409	1409
   query60	352	326	314	314
   query61	150	160	189	160
   query62	666	620	530	530
   query63	241	216	208	208
   query64	2440	809	671	671
   query65	
   query66	1751	519	384	384
   query67	29973	30002	29716	29716
   query68	
   query69	459	334	300	300
   query70	1018	981	972	972
   query71	311	270	279	270
   query72	2953	2697	2424	2424
   query73	817	790	432	432
   query74	5090	4890	4766	4766
   query75	2834	2690	2331	2331
   query76	2292	1133	792	792
   query77	412	419	344	344
   query78	13021	13011	12314	12314
   query79	1491	1022	762	762
   query80	1399	568	495	495
   query81	539	280	233	233
   query82	1232	156	124	124
   query83	355	274	249	249
   query84	256	137	117	117
   query85	929	517	441	441
   query86	455	322	336	322
   query87	3427	3331	3236	3236
   query88	3585	2704	2692	2692
   query89	437	388	337	337
   query90	1949	181	181	181
   query91	177	170	139	139
   query92	79	77	72	72
   query93	955	964	567	567
   query94	734	358	289	289
   query95	672	376	344	344
   query96	1030	754	342	342
   query97	2675	2686	2582	2582
   query98	238	229	235	229
   query99	1151	1116	990	990
   Total cold run time: 254313 ms
   Total hot run time: 170728 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4432231933

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4317073497

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4317108443

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4265945038

   # FE Regression Coverage Report
   Increment line coverage `  77.67% (80/103)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62304_eee072014d893e3257b248f16a29d6dba02a4429_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62304_eee072014d893e3257b248f16a29d6dba02a4429_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62304:
URL: https://github.com/apache/doris/pull/62304#discussion_r3212735833


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java:
##########
@@ -506,14 +546,35 @@ public Void visitArraySortBy(ArraySortBy arraySortBy, CollectorContext context)
         return visit(arraySortBy, context);
     }
 
-    // @Override
-    // public Void visitIsNull(IsNull isNull, CollectorContext context) {
-    //     if (context.accessPathBuilder.isEmpty()) {
-    //         context.setType(ColumnAccessPathType.META);
-    //         return continueCollectAccessPath(isNull.child(), context);
-    //     }
-    //     return visit(isNull, context);
-    // }
+    @Override
+    public Void visitIsNull(IsNull isNull, CollectorContext context) {
+        Expression arg = isNull.child();
+        // Skip variant sub-column paths (v['k'] IS NULL): the sub-column path is already baked
+        // into the SlotReference, so null-only access doesn't apply the same way.
+        if (arg instanceof SlotReference && ((SlotReference) arg).hasSubColPath()) {
+            return visit(isNull, context);
+        }
+        // Optimize IS NULL on nullable expressions: create a context with NULL suffix to indicate
+        // only the null flag is needed. Works for top-level columns (col IS NULL → [col, NULL])
+        // and nested access (struct_element(s, 'city') IS NULL → [s, city, NULL]).
+        // For unrecognized expressions, the default visitor resets context, safely discarding NULL.
+        if (arg.nullable() && context.accessPathBuilder.isEmpty()) {
+            CollectorContext nullContext =
+                    new CollectorContext(context.statementContext, context.bottomFilter);
+            nullContext.accessPathBuilder.addSuffix(AccessPathInfo.ACCESS_NULL);
+            return continueCollectAccessPath(arg, nullContext);

Review Comment:
   This NULL suffix is also propagated through `map_keys(map_col)` and `map_values(map_col)`, producing paths like `[map_col, KEYS, NULL]` / `[map_col, VALUES, NULL]` (the new regression asserts `map_col.KEYS.NULL`). But the nullness of `map_keys(nullable_map)` or `map_values(nullable_map)` is determined by the parent map null map, not by the key/value child column. In BE, `MapFileColumnIterator::set_access_paths()` only switches to `NULL_MAP_ONLY` when the first sub-path component after the map name is `NULL`; with `KEYS.NULL`/`VALUES.NULL` it routes the path to the key/value iterator and still reads offsets plus child data (or relies on scalar fallback). For `map_keys(map_col) IS NULL` / `IS NOT NULL`, FE should emit/use the parent `[map_col, NULL]` path, or otherwise add BE handling that treats these function-level null checks as parent-map null-map-only reads.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4412140271

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29480 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit f0cfe863011095f809f57d6db6e0a86c2de5ca1a, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	18173	3983	3890	3890
   q2	q3	10732	870	611	611
   q4	4660	456	346	346
   q5	7451	1349	1161	1161
   q6	192	172	142	142
   q7	910	938	763	763
   q8	9316	1419	1260	1260
   q9	5624	5396	5339	5339
   q10	6261	2078	1799	1799
   q11	461	266	261	261
   q12	630	406	291	291
   q13	18065	3383	2722	2722
   q14	300	286	263	263
   q15	q16	865	847	794	794
   q17	910	945	745	745
   q18	6504	5665	5507	5507
   q19	1167	1260	1051	1051
   q20	502	412	265	265
   q21	4627	2430	1935	1935
   q22	481	423	335	335
   Total cold run time: 97831 ms
   Total hot run time: 29480 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4820	4769	4810	4769
   q2	q3	4662	4800	4182	4182
   q4	2106	2198	1396	1396
   q5	4997	5005	5240	5005
   q6	193	170	149	149
   q7	2103	1805	1601	1601
   q8	3362	3053	3087	3053
   q9	8405	8475	8510	8475
   q10	4496	4482	4289	4289
   q11	613	410	386	386
   q12	692	761	507	507
   q13	3291	3633	2977	2977
   q14	311	319	274	274
   q15	q16	758	810	690	690
   q17	1477	1285	1257	1257
   q18	8029	7196	7106	7106
   q19	1170	1139	1153	1139
   q20	2248	2232	1951	1951
   q21	6103	5424	4868	4868
   q22	555	503	430	430
   Total cold run time: 60391 ms
   Total hot run time: 54504 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4414072361

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4421062098

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29467 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 83d42d9aa645228fb4135a059ce26a489db011c4, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17617	3844	3780	3780
   q2	q3	10699	862	597	597
   q4	4678	455	339	339
   q5	7447	1316	1133	1133
   q6	184	165	138	138
   q7	886	951	723	723
   q8	9302	1364	1258	1258
   q9	5550	5398	5301	5301
   q10	6328	2079	1805	1805
   q11	472	273	261	261
   q12	689	410	281	281
   q13	18190	3216	2733	2733
   q14	292	278	263	263
   q15	q16	893	870	790	790
   q17	1019	947	777	777
   q18	6344	5677	5589	5589
   q19	1204	1272	1104	1104
   q20	505	400	276	276
   q21	4637	2336	1994	1994
   q22	458	394	325	325
   Total cold run time: 97394 ms
   Total hot run time: 29467 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4737	4483	4522	4483
   q2	q3	4630	4753	4182	4182
   q4	2120	2166	1418	1418
   q5	5176	4994	5273	4994
   q6	190	167	133	133
   q7	2049	1798	1662	1662
   q8	3346	3072	3113	3072
   q9	8643	8492	8391	8391
   q10	4482	4457	4230	4230
   q11	580	426	403	403
   q12	716	772	534	534
   q13	3226	3523	2976	2976
   q14	298	299	274	274
   q15	q16	743	780	684	684
   q17	1362	1297	1247	1247
   q18	7971	7076	7074	7074
   q19	1158	1136	1155	1136
   q20	2209	2232	1989	1989
   q21	5992	5255	4750	4750
   q22	523	476	404	404
   Total cold run time: 60151 ms
   Total hot run time: 54036 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4313939384

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) Optimize I/O operations for the IS NULL predicate [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #62304:
URL: https://github.com/apache/doris/pull/62304#issuecomment-4853573094

   This PR implements the IS NULL optimization by removing the commented-out META-type `visitIsNull` and re-implementing it as an `ACCESS_NULL` path suffix (`[col, NULL]`). The removed version was:
   
       // context.setType(ColumnAccessPathType.META);
   
   Same "read-mode in the path string" direction as the OFFSET tier (#62205) — and this is the NULL half of the namespace collision I filed there: a struct field literally named `null` gets read as the NULL meta by BE's case-insensitive `StringCaseEqual`, so the struct enters `NULL_MAP_ONLY` and skips all sub-columns. Verified on a current build:
   
   ```sql
   CREATE TABLE t(id INT, s STRUCT<`null`:INT, x:INT>)
     DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES("replication_num"="1");
   INSERT INTO t VALUES(1, named_struct('null',100,'x',200));
   SELECT s.`null` FROM t;   -- pruning ON -> NULL (wrong);  OFF -> 100 (correct)
   ```
   
   Two things specific to this PR:
   1. The `stripNullSuffixPaths` comment here already documents the BE trap verbatim ("Struct/Array/Map iterators treat a leading NULL sub-path as NULL_MAP_ONLY and skip all children") — the behavior was known, but only the legitimate-meta path was guarded; the field-name entry was missed.
   2. NULL is more dangerous than OFFSET: the struct reader actually acts on `NULL_MAP_ONLY`, while it has no `OFFSET_ONLY` branch — so an `offset` field survives by luck but a `null` field does not.
   
   The functional IS NULL pruning itself tests correct (scalar / struct / variant / mixed / IS NOT NULL / WHERE pushdown all match the non-pruned results). The concern is the design direction: encoding read-mode as a path component reintroduces the namespace it then collides on. Keeping read-mode as the orthogonal `DATA`/`META` type dimension — the one this PR removed — avoids it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org