You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yujun777 (via GitHub)" <gi...@apache.org> on 2026/10/19 12:07:06 UTC

[PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

yujun777 opened a new pull request, #69286:
URL: https://github.com/apache/doris/pull/69286

   ### What problem does this PR solve?
   
   Incremental refresh of a MIN/MAX materialized view currently falls back to a
   complete refresh whenever a deleted row could be the current extreme. Deleting
   one of several rows that share the minimum value is a common case, yet the MV
   does not remember how many rows hold that extreme, so the refresh cannot tell
   that the extreme survives.
   
   This PR adds `COUNT_AT_MIN` / `COUNT_AT_MAX` and stores that count as a hidden
   IVM state column. The delta plan now emits the count at the insert-side and
   delete-side extreme. Apply keeps the old extreme while its surviving count is
   positive; only deleting the last row at the extreme still falls back to a
   complete refresh.
   
   Trace issue: https://github.com/apache/doris/issues/65418
   
   ### What changed?
   
   - Add public `COUNT_AT_MIN(expr)` / `COUNT_AT_MAX(expr)` aggregate functions.
     They return the number of input rows equal to the group's minimum/maximum,
     ignore NULLs, return 0 for empty or all-NULL input, and reject DISTINCT.
     They are equivalent to Oracle's
     `COUNT(*) KEEP (DENSE_RANK FIRST/LAST ORDER BY expr)`.
   - Add `MIN_COUNT` / `MAX_COUNT` IVM hidden state keys and hidden columns named
     in the existing `__DORIS_IVM_AGG_{ordinal}_{STATE}_COL__` format.
   - Emit insert/delete extreme counts in the delta aggregate.
   - Use the surviving extreme count in the apply guard and update the hidden
     count state.
   - Add BE/FE unit coverage and a regression suite.
   
   ### Testing
   
   - FE unit tests: `IvmAggMinProcessorTest`, `IvmAggMaxProcessorTest`,
     `IvmAggDeltaHandlerTest`, `IvmNormalizeUniqKeysMTMVTest`.
   - Regression: `test_ivm_minmax_count`, plus existing
     `test_ivm_minmax_runtime_fallback`, `test_ivm_agg_1`, `test_ivm_agg_2`, and
     `test_ivm_agg_shared_columns`.
   - SQL verification of duplicate extremes, all-NULL input, and DISTINCT
     rejection.
   - FE build/checkstyle, BE build, clang-format/check-format, and build hygiene
     pass.
   - A BE unit test for the new aggregate is included. The local BE unit-test
     target could not link in this environment because the toolchain already
     provides glibc-compatibility symbols; the main BE build succeeded.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6268544096

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6284559334

   # BE Regression && UT Coverage Report
   Increment line coverage `92.05% (81/88)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 77.23% (35815/46376) |
   | Line Coverage     | 62.47% (404199/647048) |
   | Region Coverage   | 58.95% (341456/579251) |
   | Branch Coverage   | 59.86% (157751/263526) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 merged PR #69286:
URL: https://github.com/apache/doris/pull/69286


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282323915

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 28433 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 008d08b586b9e981e19d07e8e5ca01d63f244045, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17611	5230	5036	5036
   q2	2103	313	276	276
   q3	10223	1393	844	844
   q4	4681	467	356	356
   q5	7448	832	527	527
   q6	180	187	150	150
   q7	750	815	638	638
   q8	9255	1368	1052	1052
   q9	5373	4221	4186	4186
   q10	6857	1289	1030	1030
   q11	422	247	232	232
   q12	631	395	285	285
   q13	18095	3689	3026	3026
   q14	286	273	246	246
   q15	q16	741	704	644	644
   q17	1224	752	582	582
   q18	7062	6182	6091	6091
   q19	1255	959	605	605
   q20	384	330	226	226
   q21	5330	2153	2268	2153
   q22	362	307	248	248
   Total cold run time: 100273 ms
   Total hot run time: 28433 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	5615	5627	5603	5603
   q2	702	547	524	524
   q3	5187	5360	4914	4914
   q4	1999	2147	1545	1545
   q5	5280	5136	5123	5123
   q6	235	181	146	146
   q7	2069	1826	1711	1711
   q8	2844	2456	2453	2453
   q9	7508	7527	7538	7527
   q10	3651	3570	3124	3124
   q11	578	433	378	378
   q12	663	670	504	504
   q13	3167	3541	3058	3058
   q14	290	293	271	271
   q15	q16	670	699	623	623
   q17	7723	6982	6924	6924
   q18	12974	12172	12908	12172
   q19	944	831	853	831
   q20	2125	2164	1914	1914
   q21	5632	4646	4643	4643
   q22	491	446	441	441
   Total cold run time: 70347 ms
   Total hot run time: 64429 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69286:
URL: https://github.com/apache/doris/pull/69286#discussion_r4320711849


##########
be/src/exprs/aggregate/aggregate_function_count_at_min_max.cpp:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/aggregate/aggregate_function_count_at_min_max.h"
+
+#include "common/exception.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/define_primitive_type.h"
+#include "core/data_type/primitive_type.h"
+#include "exprs/aggregate/aggregate_function_simple_factory.h"
+#include "exprs/aggregate/factory_helpers.h"
+#include "exprs/aggregate/helpers.h"
+
+namespace doris {
+
+namespace {
+
+template <template <typename> class ExtremeData>
+AggregateFunctionPtr create_aggregate_function_count_at_extreme(const String& name,
+                                                                const DataTypes& argument_types,
+                                                                const DataTypePtr& result_type,
+                                                                const bool result_is_nullable,
+                                                                const AggregateFunctionAttr& attr) {
+    assert_arity_range(name, argument_types, 1, 1);
+    switch (argument_types[0]->get_primitive_type()) {
+    case PrimitiveType::TYPE_STRING:
+    case PrimitiveType::TYPE_CHAR:
+    case PrimitiveType::TYPE_VARCHAR:
+    case PrimitiveType::TYPE_JSONB:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataString>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_DATE:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATE>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_DATETIME:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATETIME>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_DATEV2:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATEV2>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_DATETIMEV2:
+        return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+                ExtremeData<SingleValueDataFixed<TYPE_DATETIMEV2>>>>(argument_types,
+                                                                     result_is_nullable, attr);
+    case PrimitiveType::TYPE_TIMESTAMP_NS:
+        return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+                ExtremeData<SingleValueDataFixed<TYPE_TIMESTAMP_NS>>>>(argument_types,
+                                                                       result_is_nullable, attr);
+    case PrimitiveType::TYPE_TIMESTAMPTZ:
+        return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+                ExtremeData<SingleValueDataFixed<TYPE_TIMESTAMPTZ>>>>(argument_types,
+                                                                      result_is_nullable, attr);
+    case PrimitiveType::TYPE_TIMEV2:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_TIMEV2>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_IPV4:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_IPV4>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_IPV6:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_IPV6>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_UUID:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_UUID>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_BOOLEAN:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_BOOLEAN>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_TINYINT:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_TINYINT>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_SMALLINT:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_SMALLINT>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_INT:
+        return creator_without_type::create_unary_arguments<

Review Comment:
   [P2] Preserve zero results in the foreach variants
   
   For an `ARRAY<INT>` column `a` containing `[1,NULL]` and `[2,NULL]`, both `count_at_min_foreach(a)` and `count_at_max_foreach(a)` produce `[1,NULL]` instead of `[1,0]`. The FE exposes these combinators automatically, and `foreachv2` requests `result_is_nullable=true` for its nested function. Forwarding that flag through `create_unary_arguments` installs a nullable-result adapter for the nullable array elements; an all-NULL position then emits NULL without calling the count state's zero-result insertion. Both old and V2 nullable adapters take this path. Use the nonnullable-result unary creator in all type branches (or otherwise preserve the nested aggregate's nonnullable semantics), and add an all-NULL-element regression for both variants.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281835541

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6283129100

   # FE UT Coverage Report
   Increment line coverage `  71.43% (90/126)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6266817858

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6271286998

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.32 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 87a8d552ab57c552322af4ce4a77ca290d6829ac, data reload: false
   
   query1	0.01	0.01	0.00
   query2	0.09	0.05	0.05
   query3	0.26	0.09	0.08
   query4	1.60	0.10	0.10
   query5	0.23	0.25	0.23
   query6	1.17	0.69	0.66
   query7	0.04	0.01	0.01
   query8	0.05	0.04	0.03
   query9	0.38	0.33	0.36
   query10	0.60	0.61	0.57
   query11	0.21	0.15	0.14
   query12	0.19	0.15	0.15
   query13	0.49	0.48	0.48
   query14	1.03	1.02	1.01
   query15	0.63	0.60	0.59
   query16	0.34	0.35	0.35
   query17	1.14	1.18	1.15
   query18	0.21	0.19	0.19
   query19	2.09	2.05	2.00
   query20	0.02	0.01	0.02
   query21	15.48	0.20	0.14
   query22	4.80	0.06	0.05
   query23	16.25	0.30	0.11
   query24	3.11	0.62	0.49
   query25	0.09	0.05	0.06
   query26	1.59	0.22	0.17
   query27	0.05	0.04	0.05
   query28	3.48	0.77	0.35
   query29	12.50	4.10	3.22
   query30	0.28	0.14	0.10
   query31	2.77	0.55	0.32
   query32	3.22	0.60	0.51
   query33	3.22	3.15	3.22
   query34	15.51	4.11	3.39
   query35	3.34	3.34	3.35
   query36	0.57	0.44	0.42
   query37	0.10	0.06	0.07
   query38	0.05	0.04	0.03
   query39	0.04	0.03	0.03
   query40	0.17	0.14	0.14
   query41	0.09	0.03	0.03
   query42	0.03	0.02	0.03
   query43	0.03	0.03	0.04
   Total cold run time: 97.55 s
   Total hot run time: 24.32 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6271201127

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 150262 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 87a8d552ab57c552322af4ce4a77ca290d6829ac, data reload: false
   
   query5	4299	579	433	433
   query6	426	223	191	191
   query7	4809	440	228	228
   query8	351	170	151	151
   query9	8762	3035	3037	3035
   query10	471	303	248	248
   query11	6001	3843	3569	3569
   query12	143	95	88	88
   query13	1255	466	327	327
   query14	6155	3908	3621	3621
   query14_1	3360	3399	3376	3376
   query15	209	197	180	180
   query16	933	424	404	404
   query17	885	701	528	528
   query18	2440	436	312	312
   query19	212	173	143	143
   query20	94	82	82	82
   query21	212	134	109	109
   query22	12979	13034	12783	12783
   query23	13424	12911	12441	12441
   query23_1	12607	12502	12517	12502
   query24	7383	809	459	459
   query24_1	469	472	466	466
   query25	531	415	368	368
   query26	1262	253	145	145
   query27	2770	438	269	269
   query28	4638	1910	1935	1910
   query29	1553	603	462	462
   query30	298	219	185	185
   query31	1012	929	829	829
   query32	139	94	95	94
   query33	543	313	247	247
   query34	886	853	495	495
   query35	728	796	651	651
   query36	821	814	729	729
   query37	126	98	84	84
   query38	1745	1749	1714	1714
   query39	791	800	738	738
   query39_1	731	724	744	724
   query40	215	119	97	97
   query41	73	68	67	67
   query42	87	86	82	82
   query43	338	362	316	316
   query44	1293	677	683	677
   query45	179	173	168	168
   query46	822	964	563	563
   query47	2801	3028	2852	2852
   query48	296	307	250	250
   query49	569	420	288	288
   query50	657	262	211	211
   query51	10437	10261	10590	10261
   query52	76	79	75	75
   query53	182	214	150	150
   query54	231	195	177	177
   query55	69	75	61	61
   query56	237	212	217	212
   query57	1640	1598	1554	1554
   query58	292	256	256	256
   query59	2148	2196	1928	1928
   query60	288	237	210	210
   query61	163	147	142	142
   query62	384	338	276	276
   query63	192	160	155	155
   query64	2718	974	766	766
   query65	3289	3241	3251	3241
   query66	1775	413	297	297
   query67	20421	20138	19972	19972
   query68	3026	992	591	591
   query69	383	274	244	244
   query70	860	768	735	735
   query71	283	237	210	210
   query72	2705	2572	2267	2267
   query73	508	525	287	287
   query74	4586	4462	4326	4326
   query75	2312	2293	1920	1920
   query76	2299	994	589	589
   query77	341	364	272	272
   query78	9692	9664	9146	9146
   query79	984	841	516	516
   query80	1254	427	333	333
   query81	587	327	264	264
   query82	580	142	104	104
   query83	306	200	178	178
   query84	308	115	101	101
   query85	859	414	371	371
   query86	402	239	229	229
   query87	2016	1939	1808	1808
   query88	3612	2654	2628	2628
   query89	334	284	241	241
   query90	1933	174	162	162
   query91	163	149	124	124
   query92	95	86	91	86
   query93	1012	992	570	570
   query94	645	293	250	250
   query95	569	405	297	297
   query96	659	478	228	228
   query97	2452	2452	2377	2377
   query98	166	151	148	148
   query99	762	768	648	648
   Total cold run time: 234685 ms
   Total hot run time: 150262 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6287116624

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281385633

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282048300

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6269868681

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 28306 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit c5d12fabf6c91e627068dd92592f5b5fd2aa1e62, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17642	5179	5021	5021
   q2	2111	288	251	251
   q3	10281	1348	835	835
   q4	4679	453	335	335
   q5	7427	800	539	539
   q6	181	178	149	149
   q7	743	803	595	595
   q8	9252	1262	1056	1056
   q9	5440	4203	4204	4203
   q10	6840	1282	1041	1041
   q11	438	250	230	230
   q12	639	399	286	286
   q13	18084	3584	2973	2973
   q14	272	273	242	242
   q15	q16	708	709	658	658
   q17	1193	750	584	584
   q18	6984	6176	6115	6115
   q19	1088	966	601	601
   q20	383	333	226	226
   q21	5085	2110	2311	2110
   q22	341	293	256	256
   Total cold run time: 99811 ms
   Total hot run time: 28306 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	5625	5635	5682	5635
   q2	660	590	581	581
   q3	5354	5504	5026	5026
   q4	2052	2162	1551	1551
   q5	5337	5122	5609	5122
   q6	248	195	156	156
   q7	2304	2036	1863	1863
   q8	2974	2720	2694	2694
   q9	8049	7904	7927	7904
   q10	3738	3726	3468	3468
   q11	609	453	411	411
   q12	746	724	533	533
   q13	3495	3794	3300	3300
   q14	323	348	308	308
   q15	q16	706	705	658	658
   q17	8134	7423	6831	6831
   q18	13017	12158	12830	12158
   q19	922	790	787	787
   q20	2150	2142	1876	1876
   q21	5596	4572	4627	4572
   q22	476	441	390	390
   Total cold run time: 72515 ms
   Total hot run time: 65824 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6266815938

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6272583103

   # FE Regression Coverage Report
   Increment line coverage `  79.37% (100/126)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6272792414

   # FE Regression Coverage Report
   Increment line coverage `  79.37% (100/126)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282505041

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.36 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 008d08b586b9e981e19d07e8e5ca01d63f244045, data reload: false
   
   query1	0.01	0.01	0.01
   query2	0.09	0.05	0.05
   query3	0.26	0.10	0.09
   query4	1.60	0.10	0.09
   query5	0.23	0.26	0.24
   query6	1.16	0.71	0.68
   query7	0.04	0.00	0.00
   query8	0.05	0.04	0.03
   query9	0.41	0.34	0.35
   query10	0.57	0.56	0.56
   query11	0.20	0.14	0.15
   query12	0.18	0.15	0.15
   query13	0.50	0.49	0.49
   query14	1.04	1.00	1.02
   query15	0.64	0.63	0.61
   query16	0.31	0.35	0.39
   query17	1.13	1.12	1.13
   query18	0.19	0.19	0.19
   query19	2.06	1.99	1.99
   query20	0.02	0.01	0.01
   query21	15.49	0.19	0.14
   query22	4.97	0.06	0.05
   query23	16.24	0.32	0.13
   query24	3.07	0.49	0.44
   query25	0.08	0.06	0.05
   query26	0.99	0.21	0.16
   query27	0.03	0.03	0.04
   query28	3.47	0.77	0.37
   query29	12.49	4.15	3.31
   query30	0.29	0.12	0.10
   query31	2.77	0.56	0.31
   query32	3.22	0.60	0.50
   query33	3.15	3.32	3.18
   query34	15.46	4.00	3.40
   query35	3.34	3.29	3.36
   query36	0.54	0.48	0.41
   query37	0.09	0.07	0.06
   query38	0.05	0.04	0.04
   query39	0.04	0.03	0.03
   query40	0.17	0.14	0.13
   query41	0.08	0.03	0.03
   query42	0.03	0.03	0.03
   query43	0.04	0.03	0.04
   Total cold run time: 96.79 s
   Total hot run time: 24.36 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281835214

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6271027430

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 28165 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 87a8d552ab57c552322af4ce4a77ca290d6829ac, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17587	5103	5005	5005
   q2	2109	304	245	245
   q3	10258	1422	822	822
   q4	4693	459	338	338
   q5	7424	790	532	532
   q6	179	188	154	154
   q7	731	784	585	585
   q8	9241	1343	1039	1039
   q9	5431	4172	4162	4162
   q10	6813	1277	1010	1010
   q11	430	244	221	221
   q12	646	394	277	277
   q13	18105	3539	2966	2966
   q14	272	270	245	245
   q15	q16	701	719	654	654
   q17	1182	751	578	578
   q18	6965	6152	6115	6115
   q19	1113	949	606	606
   q20	387	327	226	226
   q21	5009	2132	2312	2132
   q22	342	295	253	253
   Total cold run time: 99618 ms
   Total hot run time: 28165 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	5593	5576	5566	5566
   q2	674	574	548	548
   q3	5223	5359	4949	4949
   q4	1980	2151	1559	1559
   q5	5251	5158	5425	5158
   q6	250	195	146	146
   q7	2335	2071	1855	1855
   q8	3033	2669	2717	2669
   q9	8074	8053	7841	7841
   q10	3745	3653	3238	3238
   q11	611	484	470	470
   q12	682	708	522	522
   q13	3476	3915	3266	3266
   q14	331	340	301	301
   q15	q16	730	706	650	650
   q17	8257	7185	6855	6855
   q18	12947	12227	12836	12227
   q19	898	798	775	775
   q20	2129	2177	1909	1909
   q21	5546	4554	4645	4554
   q22	481	463	399	399
   Total cold run time: 72246 ms
   Total hot run time: 65457 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6286939023

   # FE Regression Coverage Report
   Increment line coverage `  60.34% (108/179)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6270074647

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 150451 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit c5d12fabf6c91e627068dd92592f5b5fd2aa1e62, data reload: false
   
   query5	4325	580	436	436
   query6	438	205	194	194
   query7	4815	455	227	227
   query8	311	168	156	156
   query9	8790	3124	3113	3113
   query10	486	304	252	252
   query11	5961	3838	3573	3573
   query12	146	92	87	87
   query13	1259	440	322	322
   query14	6141	3967	3650	3650
   query14_1	3443	3429	3466	3429
   query15	208	194	178	178
   query16	966	441	409	409
   query17	855	660	553	553
   query18	2415	412	315	315
   query19	190	171	132	132
   query20	88	94	87	87
   query21	206	126	110	110
   query22	12984	12936	12852	12852
   query23	13410	12775	12395	12395
   query23_1	12461	12538	12431	12431
   query24	7380	818	461	461
   query24_1	480	476	462	462
   query25	535	424	365	365
   query26	1256	254	142	142
   query27	2760	438	272	272
   query28	4653	1934	1928	1928
   query29	1565	604	477	477
   query30	302	224	183	183
   query31	1016	935	801	801
   query32	151	95	94	94
   query33	531	304	248	248
   query34	954	846	495	495
   query35	721	773	622	622
   query36	805	820	721	721
   query37	120	92	79	79
   query38	1747	1701	1660	1660
   query39	786	798	744	744
   query39_1	730	727	740	727
   query40	213	123	92	92
   query41	68	63	62	62
   query42	84	84	81	81
   query43	344	371	311	311
   query44	1277	673	679	673
   query45	176	173	169	169
   query46	841	941	559	559
   query47	3012	3051	2889	2889
   query48	296	302	214	214
   query49	578	389	276	276
   query50	667	267	208	208
   query51	10548	10237	10160	10160
   query52	79	78	70	70
   query53	181	203	153	153
   query54	232	189	176	176
   query55	75	71	64	64
   query56	294	230	208	208
   query57	1635	1627	1558	1558
   query58	287	255	251	251
   query59	2169	2208	1965	1965
   query60	295	243	227	227
   query61	146	149	152	149
   query62	387	328	292	292
   query63	187	158	160	158
   query64	2735	917	769	769
   query65	3304	3241	3268	3241
   query66	1771	437	292	292
   query67	20391	19958	20254	19958
   query68	2975	1005	603	603
   query69	377	291	250	250
   query70	821	788	783	783
   query71	288	213	203	203
   query72	2752	2579	2343	2343
   query73	525	518	298	298
   query74	4603	4500	4375	4375
   query75	2337	2285	1900	1900
   query76	2321	1001	619	619
   query77	356	376	273	273
   query78	9861	9698	9133	9133
   query79	997	838	527	527
   query80	690	448	361	361
   query81	555	313	272	272
   query82	575	139	107	107
   query83	295	203	182	182
   query84	311	128	100	100
   query85	948	415	362	362
   query86	336	236	234	234
   query87	2005	1977	1804	1804
   query88	3612	2656	2618	2618
   query89	331	280	242	242
   query90	1831	166	174	166
   query91	165	149	128	128
   query92	95	89	81	81
   query93	995	979	577	577
   query94	478	312	261	261
   query95	582	385	301	301
   query96	671	478	228	228
   query97	2426	2455	2336	2336
   query98	170	150	146	146
   query99	777	760	651	651
   Total cold run time: 234517 ms
   Total hot run time: 150451 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6270177849

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.09 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit c5d12fabf6c91e627068dd92592f5b5fd2aa1e62, data reload: false
   
   query1	0.00	0.00	0.01
   query2	0.10	0.05	0.05
   query3	0.26	0.09	0.08
   query4	1.61	0.10	0.10
   query5	0.23	0.24	0.24
   query6	1.18	0.70	0.67
   query7	0.04	0.01	0.01
   query8	0.05	0.04	0.04
   query9	0.41	0.36	0.36
   query10	0.60	0.57	0.59
   query11	0.20	0.14	0.14
   query12	0.19	0.14	0.15
   query13	0.48	0.49	0.50
   query14	1.04	1.02	1.00
   query15	0.62	0.59	0.60
   query16	0.36	0.34	0.30
   query17	1.15	1.09	1.14
   query18	0.20	0.20	0.18
   query19	2.03	2.04	1.98
   query20	0.02	0.01	0.02
   query21	15.50	0.19	0.14
   query22	4.83	0.06	0.05
   query23	16.38	0.32	0.12
   query24	2.86	0.48	0.41
   query25	0.10	0.06	0.04
   query26	1.22	0.21	0.15
   query27	0.04	0.05	0.03
   query28	3.29	0.82	0.36
   query29	12.48	4.09	3.24
   query30	0.27	0.14	0.10
   query31	2.77	0.56	0.32
   query32	3.24	0.58	0.51
   query33	3.28	3.13	3.32
   query34	15.43	3.93	3.36
   query35	3.33	3.32	3.34
   query36	0.57	0.45	0.43
   query37	0.09	0.07	0.06
   query38	0.05	0.03	0.04
   query39	0.04	0.03	0.03
   query40	0.17	0.14	0.14
   query41	0.08	0.03	0.02
   query42	0.03	0.03	0.03
   query43	0.04	0.04	0.03
   Total cold run time: 96.86 s
   Total hot run time: 24.09 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69286:
URL: https://github.com/apache/doris/pull/69286#discussion_r4320918485


##########
be/src/exprs/aggregate/aggregate_function_count_at_min_max.cpp:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/aggregate/aggregate_function_count_at_min_max.h"
+
+#include "common/exception.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/define_primitive_type.h"
+#include "core/data_type/primitive_type.h"
+#include "exprs/aggregate/aggregate_function_simple_factory.h"
+#include "exprs/aggregate/factory_helpers.h"
+#include "exprs/aggregate/helpers.h"
+
+namespace doris {
+
+namespace {
+
+template <template <typename> class ExtremeData>
+AggregateFunctionPtr create_aggregate_function_count_at_extreme(const String& name,
+                                                                const DataTypes& argument_types,
+                                                                const DataTypePtr& result_type,
+                                                                const bool result_is_nullable,
+                                                                const AggregateFunctionAttr& attr) {
+    assert_arity_range(name, argument_types, 1, 1);
+    switch (argument_types[0]->get_primitive_type()) {
+    case PrimitiveType::TYPE_STRING:
+    case PrimitiveType::TYPE_CHAR:
+    case PrimitiveType::TYPE_VARCHAR:
+    case PrimitiveType::TYPE_JSONB:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataString>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_DATE:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATE>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_DATETIME:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATETIME>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_DATEV2:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATEV2>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_DATETIMEV2:
+        return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+                ExtremeData<SingleValueDataFixed<TYPE_DATETIMEV2>>>>(argument_types,
+                                                                     result_is_nullable, attr);
+    case PrimitiveType::TYPE_TIMESTAMP_NS:
+        return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+                ExtremeData<SingleValueDataFixed<TYPE_TIMESTAMP_NS>>>>(argument_types,
+                                                                       result_is_nullable, attr);
+    case PrimitiveType::TYPE_TIMESTAMPTZ:
+        return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+                ExtremeData<SingleValueDataFixed<TYPE_TIMESTAMPTZ>>>>(argument_types,
+                                                                      result_is_nullable, attr);
+    case PrimitiveType::TYPE_TIMEV2:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_TIMEV2>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_IPV4:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_IPV4>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_IPV6:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_IPV6>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_UUID:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_UUID>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_BOOLEAN:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_BOOLEAN>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_TINYINT:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_TINYINT>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_SMALLINT:
+        return creator_without_type::create_unary_arguments<
+                AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_SMALLINT>>>>(
+                argument_types, result_is_nullable, attr);
+    case PrimitiveType::TYPE_INT:
+        return creator_without_type::create_unary_arguments<

Review Comment:
   Fixed in 008d08b586b. The nested count_at_min/count_at_max now uses create_unary_arguments_return_not_nullable, so the foreach wrapper keeps the non-nullable zero result instead of emitting NULL. I added an all-NULL array position regression for both count_at_min_foreach and count_at_max_foreach, under the default V2 and the old nullable adapter; both now return [1,0].



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282033674

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.13 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit fe394a7d126524423d42188559245ad083f7f3c6, data reload: false
   
   query1	0.00	0.00	0.00
   query2	0.09	0.04	0.05
   query3	0.26	0.09	0.09
   query4	1.61	0.11	0.11
   query5	0.25	0.23	0.25
   query6	1.18	0.70	0.68
   query7	0.05	0.01	0.01
   query8	0.06	0.04	0.04
   query9	0.39	0.35	0.33
   query10	0.56	0.56	0.59
   query11	0.20	0.15	0.14
   query12	0.18	0.15	0.15
   query13	0.50	0.48	0.50
   query14	1.05	1.01	1.02
   query15	0.62	0.60	0.60
   query16	0.36	0.35	0.32
   query17	1.14	1.15	1.11
   query18	0.20	0.20	0.20
   query19	2.12	1.95	1.99
   query20	0.02	0.01	0.01
   query21	15.49	0.18	0.13
   query22	4.91	0.06	0.05
   query23	16.29	0.32	0.12
   query24	3.15	0.55	0.33
   query25	0.11	0.05	0.06
   query26	0.92	0.23	0.15
   query27	0.05	0.04	0.04
   query28	2.51	0.73	0.36
   query29	12.50	4.09	3.25
   query30	0.28	0.13	0.12
   query31	2.77	0.55	0.32
   query32	3.23	0.59	0.50
   query33	3.12	3.30	3.18
   query34	15.44	3.99	3.38
   query35	3.33	3.34	3.32
   query36	0.54	0.46	0.42
   query37	0.10	0.07	0.07
   query38	0.04	0.03	0.04
   query39	0.04	0.03	0.03
   query40	0.18	0.14	0.14
   query41	0.09	0.02	0.03
   query42	0.04	0.03	0.04
   query43	0.04	0.03	0.04
   Total cold run time: 96.01 s
   Total hot run time: 24.13 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69286:
URL: https://github.com/apache/doris/pull/69286#discussion_r4313715856


##########
be/test/exprs/aggregate/agg_count_at_min_max_test.cpp:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include <cstring>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "core/column/column.h"
+#include "core/column/column_string.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+#include "exprs/aggregate/aggregate_function_simple_factory.h"
+#include "exprs/aggregate/single_value_data.h"
+
+namespace doris {
+
+void register_aggregate_function_count_at_min_max(AggregateFunctionSimpleFactory& factory);
+
+namespace {
+
+AggregateFunctionPtr create_function(AggregateFunctionSimpleFactory& factory,
+                                     const std::string& name, const DataTypes& argument_types) {
+    auto function = factory.get(name, argument_types, nullptr, false, -1);
+    EXPECT_NE(function, nullptr);
+    return function;
+}
+
+void aggregate_int32(IAggregateFunction* function, Arena& arena, const IColumn& column,
+                     int64_t expected) {
+    std::unique_ptr<char[]> memory(new char[function->size_of_data()]);
+    AggregateDataPtr place = memory.get();
+    function->create(place);
+    const IColumn* columns[1] = {&column};
+    for (size_t row = 0; row < column.size(); ++row) {
+        function->add(place, columns, row, arena);
+    }
+    ColumnInt64 result;
+    function->insert_result_into(place, result);
+    EXPECT_EQ(expected, result.get_element(0));
+    function->destroy(place);
+}
+
+} // namespace
+
+TEST(CountAtMinMaxTest, Int32) {
+    AggregateFunctionSimpleFactory factory;
+    register_aggregate_function_count_at_min_max(factory);
+    DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+
+    auto column = ColumnInt32::create();
+    for (int value : {1, 2, 1, 3, 1, 2}) {
+        column->insert_value(value);
+    }
+
+    Arena arena;
+    auto count_at_min = create_function(factory, "count_at_min", data_types);
+    aggregate_int32(count_at_min.get(), arena, *column, 3);
+
+    auto count_at_max = create_function(factory, "count_at_max", data_types);
+    aggregate_int32(count_at_max.get(), arena, *column, 1);
+}
+
+TEST(CountAtMinMaxTest, EmptyInput) {
+    AggregateFunctionSimpleFactory factory;
+    register_aggregate_function_count_at_min_max(factory);
+    DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+    Arena arena;
+
+    auto column = ColumnInt32::create();
+    for (const auto& name : {"count_at_min", "count_at_max"}) {
+        auto function = create_function(factory, name, data_types);
+        aggregate_int32(function.get(), arena, *column, 0);
+    }
+}
+
+TEST(CountAtMinMaxTest, Merge) {
+    AggregateFunctionSimpleFactory factory;
+    register_aggregate_function_count_at_min_max(factory);
+    DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+    Arena arena;
+
+    auto first = ColumnInt32::create();
+    auto second = ColumnInt32::create();
+    for (int value : {5, 1, 5}) {
+        first->insert_value(value);
+    }
+    for (int value : {5, 1, 9}) {
+        second->insert_value(value);
+    }
+
+    auto function = create_function(factory, "count_at_min", data_types);
+    std::unique_ptr<char[]> memory(new char[function->size_of_data() * 2]);
+    AggregateDataPtr lhs = memory.get();
+    AggregateDataPtr rhs = memory.get() + function->size_of_data();
+    function->create(lhs);
+    function->create(rhs);
+
+    const IColumn* first_columns[1] = {first.get()};
+    const IColumn* second_columns[1] = {second.get()};
+    for (size_t row = 0; row < first->size(); ++row) {
+        function->add(lhs, first_columns, row, arena);
+    }
+    for (size_t row = 0; row < second->size(); ++row) {
+        function->add(rhs, second_columns, row, arena);
+    }
+    function->merge(lhs, rhs, arena);
+
+    ColumnInt64 result;
+    function->insert_result_into(lhs, result);
+    EXPECT_EQ(5, result.get_element(0));

Review Comment:
   [P2] Expect two minima in the merge fixture
   
   This fixture selects `count_at_min` over partial inputs `{5,1,5}` and `{5,1,9}`. Each partial has minimum 1 with count 1, so merging equal extrema correctly returns 2. `EXPECT_EQ(5, ...)` therefore fails deterministically with the implementation in this PR. Expect 2, or change the fixture to contain five occurrences of the global minimum.



##########
regression-test/suites/mtmv_p0/ivm/test_ivm_minmax_count.groovy:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.awaitility.Awaitility
+import static java.util.concurrent.TimeUnit.SECONDS
+
+suite("test_ivm_minmax_count") {
+    // The base-table queries below assert the same rows as the MV, so transparent rewrite must be off.
+    sql """set enable_materialized_view_rewrite = false;"""
+    sql """set enable_dml_materialized_view_rewrite = false;"""
+    sql """drop materialized view if exists ivm_mm_count_mv"""
+    sql """drop table if exists ivm_mm_count_t"""
+
+    sql """
+        CREATE TABLE ivm_mm_count_t (
+            id BIGINT NOT NULL,
+            category VARCHAR(16) NOT NULL,
+            score BIGINT
+        )
+        UNIQUE KEY(id)
+        DISTRIBUTED BY HASH(id) BUCKETS 1
+        PROPERTIES (
+            "replication_num" = "1",
+            "enable_unique_key_merge_on_write" = "true",
+            "binlog.enable" = "true",
+            "binlog.format" = "ROW",
+            "binlog.need_historical_value" = "true"
+        )
+    """
+
+    // category a: min=10 twice, max=30 once
+    // category b: min=5 three times, max=5
+    sql """INSERT INTO ivm_mm_count_t VALUES
+        (1, 'a', 10), (2, 'a', 10), (3, 'a', 30),
+        (4, 'b', 5),  (5, 'b', 5),  (6, 'b', 5)"""
+
+    sql """
+        CREATE MATERIALIZED VIEW ivm_mm_count_mv
+        BUILD DEFERRED REFRESH INCREMENTAL FALLBACK ON MANUAL
+        KEY(category)
+        DISTRIBUTED BY HASH(category) BUCKETS 1
+        PROPERTIES ("replication_num" = "1")
+        AS SELECT category, MIN(score) AS min_score, MAX(score) AS max_score, COUNT(*) AS row_count
+        FROM ivm_mm_count_t
+        GROUP BY category
+    """
+
+    def assertNoFallback = { task ->
+        def reason = task.IvmFallbackReason
+        assertTrue(reason == null || reason.toString() == "\\N",
+                "unexpected IVM fallback: ${reason}")
+    }
+
+    def latestTask = {
+        def taskResult
+        Awaitility.await().atMost(300, SECONDS).pollInterval(2, SECONDS).until({
+            taskResult = sql_return_maparray("""
+                SELECT Status, RefreshMode, IvmFallbackReason, ErrorMsg
+                FROM tasks('type'='mv')
+                WHERE MvDatabaseName = '${context.dbName}'
+                  AND MvName = 'ivm_mm_count_mv'
+                ORDER BY CreateTime DESC, TaskId DESC LIMIT 1
+            """)
+            return !taskResult.isEmpty()
+                    && taskResult[0].Status.toString() != 'PENDING'
+                    && taskResult[0].Status.toString() != 'RUNNING'

Review Comment:
   [P2] Wait for the submitted refresh task before checking fallback
   
   This predicate accepts any latest terminal row without checking its TaskId. During `MTMVTask.onSuccess`, the current task is removed from the running list before `after()` publishes it into MV history, so `tasks()` can temporarily return the preceding completed refresh. At the last-min deletion, this helper can then return the previous INCREMENTAL task and fail the expected COMPLETE assertion; after the smaller insert, it can return the previous fallback task and fail `assertNoFallback`. `Suite.pollMTMVTaskTerminal` already handles this exact gap. Reuse that wait logic and inspect the same completed TaskId, or capture the previous TaskId and wait for a distinct terminal task.



##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/agg/IvmAggExtremalProcessor.java:
##########
@@ -109,48 +135,94 @@ public void appendApplyExpressions(IvmAggTarget target, IvmAggApplyContext apply
         IvmAggExpressionBuilder ctx = applyContext.expressions();
         Slot oldExtreme = applyContext.rawMvSlot(target.getValueStateColumnName());
         Expression deltaInsert = applyContext.deltaSlotValue(target, functionKind);
-        Expression deltaDel = applyContext.deltaSlotValue(target, deleteSlotRef(target));
-        Expression newCount = applyContext.buildNewHiddenCount(target);
-
+        Expression deltaDelete = applyContext.deltaSlotValue(target, deleteSlotRef(target));
+        Expression deltaInsertCount = applyContext.deltaSlotValue(target, insertCountSlotRef(target));
+        Expression deltaDeleteCount = applyContext.deltaSlotValue(target, deleteCountSlotRef(target));
+
+        Slot oldCountSlot = Objects.requireNonNull(target.getHiddenStateSlot(countStateKey),
+                "missing extreme count state for " + target);
+        Expression oldCount = ctx.zeroIfNull(applyContext.rawMvSlot(oldCountSlot.getName()));
+        Expression newTotalCount = applyContext.buildNewHiddenCount(target);
+        Expression zero = new BigIntLiteral(0L);
+
+        // Keep every predicate non-NULL: an empty insert/delete side must read as false, not
+        // as unknown, so the guard below is a plain boolean.
+        Expression insertedPresent = new Not(new IsNull(deltaInsert));
+        Expression deletedPresent = new Not(new IsNull(deltaDelete));
+        Expression insertedBetter = new And(insertedPresent, insertedIsBetter(deltaInsert, oldExtreme));
+        Expression insertedEqual = new And(insertedPresent, new EqualTo(deltaInsert, oldExtreme));
+        Expression deletedEqual = new And(deletedPresent, new EqualTo(deltaDelete, oldExtreme));
+        Expression survivingOldCount = new Subtract(
+                new Add(oldCount, new If(insertedEqual, deltaInsertCount, zero)),
+                new If(deletedEqual, deltaDeleteCount, zero));
+
+        // The old extreme survives when no delete is below it and its remaining count is
+        // positive. An inserted value that beats the old extreme also makes the merge
+        // derivable without a complete refresh.
         Expression guardCond = new Or(ImmutableList.of(
-                new EqualTo(newCount, new BigIntLiteral(0L)),
-                new IsNull(deltaDel),
+                new EqualTo(newTotalCount, zero),
                 new IsNull(oldExtreme),
-                deletedExtremeKeepsOldValueValid(deltaDel, oldExtreme, deltaInsert)
-        ));
+                new IsNull(deltaDelete),
+                new And(
+                        new And(deletedPresent,
+                                deletedExtremeIsAtLeastOld(deltaDelete, oldExtreme)),
+                        new Or(ImmutableList.of(
+                                insertedBetter,
+                                new GreaterThan(survivingOldCount, zero))))));
         Expression guard = new AssertTrue(guardCond, new StringLiteral(fallbackMessage()));
+
         Expression newExtremeRaw = new CaseWhen(
                 ImmutableList.of(
-                        new WhenClause(new EqualTo(newCount, new BigIntLiteral(0L)),
+                        new WhenClause(new EqualTo(newTotalCount, zero),
                                 new NullLiteral(oldExtreme.getDataType())),
                         new WhenClause(new IsNull(oldExtreme), deltaInsert),
-                        new WhenClause(new IsNull(deltaInsert), oldExtreme)
-                ),
-                mergeOldAndInsertedExtreme(oldExtreme, deltaInsert)
-        );
+                        new WhenClause(insertedBetter, deltaInsert)),
+                oldExtreme);
         Expression newExtremeGuarded = new If(guard, newExtremeRaw,
                 new NullLiteral(newExtremeRaw.getDataType()));
 
-        applyContext.putFinalExpression(target, target.getHiddenStateSlot(IvmAggStateKey.COUNT).getName(), newCount);
+        Expression newCountRaw = new CaseWhen(
+                ImmutableList.of(
+                        new WhenClause(new EqualTo(newTotalCount, zero), zero),
+                        new WhenClause(new IsNull(oldExtreme), deltaInsertCount),

Review Comment:
   [P1] Subtract matching deletes when initializing the extreme count
   
   When `oldExtreme` is NULL, persisting `deltaInsertCount` ignores canceled delete contributions. A reachable plan is `Aggregate(MIN(L.v)) -> InnerJoin(L.k = R.k)`: initially L has `(k=0,v=NULL)` and R has keys 0,1,2,3. Before refresh, insert L rows `(1,1),(2,1),(3,2)` and delete R key 1. The join rewrite emits `+{1,1,2}` from `leftDelta JOIN rightPre` and `-{1}` from `leftPost JOIN rightDelta`. The resulting MIN is correctly 1, but this branch stores MIN_COUNT=2 instead of 1. On the next refresh, deleting L key 2 leaves only non-NULL value 2; `survivingOldCount=2-1` stays positive, so the guard silently retains MIN=1 instead of falling back. This later wrong result is introduced by the new count state; the previous guard would reject that boundary deletion. Initialize the count from net contributions at the selected extreme and preserve fallback when its survival cannot be proved. MAX has the symmetric failure.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281849996

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 28259 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit fe394a7d126524423d42188559245ad083f7f3c6, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17603	5170	5030	5030
   q2	2103	324	253	253
   q3	10236	1365	797	797
   q4	4683	465	332	332
   q5	7421	792	526	526
   q6	179	186	152	152
   q7	760	795	595	595
   q8	9248	1312	1128	1128
   q9	5403	4174	4176	4174
   q10	6823	1279	1039	1039
   q11	424	246	226	226
   q12	639	392	274	274
   q13	18105	3572	2962	2962
   q14	275	269	245	245
   q15	q16	724	712	645	645
   q17	1199	742	582	582
   q18	6982	6170	6108	6108
   q19	1092	986	606	606
   q20	366	326	225	225
   q21	4974	2098	2301	2098
   q22	350	293	262	262
   Total cold run time: 99589 ms
   Total hot run time: 28259 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	5588	5573	5595	5573
   q2	689	587	510	510
   q3	5229	5395	4906	4906
   q4	1976	2178	1527	1527
   q5	5228	5137	5417	5137
   q6	245	192	143	143
   q7	2331	2057	1839	1839
   q8	3002	2657	2660	2657
   q9	7973	7889	8026	7889
   q10	3657	3710	3226	3226
   q11	786	501	428	428
   q12	721	658	538	538
   q13	3482	3842	3266	3266
   q14	331	332	282	282
   q15	q16	714	723	643	643
   q17	8164	7315	6868	6868
   q18	13008	12185	12849	12185
   q19	907	788	784	784
   q20	2133	2130	1922	1922
   q21	5510	4631	4674	4631
   q22	517	458	417	417
   Total cold run time: 72191 ms
   Total hot run time: 65371 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281385984

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281973497

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 150979 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit fe394a7d126524423d42188559245ad083f7f3c6, data reload: false
   
   query5	4301	583	470	470
   query6	422	226	197	197
   query7	4798	473	225	225
   query8	320	166	152	152
   query9	8762	3131	3165	3131
   query10	486	319	264	264
   query11	6031	3830	3574	3574
   query12	149	90	88	88
   query13	1248	460	325	325
   query14	6145	3956	3670	3670
   query14_1	3478	3436	3452	3436
   query15	211	197	177	177
   query16	941	436	410	410
   query17	866	665	542	542
   query18	2419	435	319	319
   query19	203	163	134	134
   query20	85	84	83	83
   query21	208	141	111	111
   query22	12971	12974	12972	12972
   query23	13348	12779	12320	12320
   query23_1	12531	12534	12550	12534
   query24	7362	826	461	461
   query24_1	483	474	481	474
   query25	533	424	356	356
   query26	1258	252	138	138
   query27	2776	446	263	263
   query28	4635	1945	1929	1929
   query29	1587	609	459	459
   query30	299	217	186	186
   query31	1018	931	803	803
   query32	148	97	95	95
   query33	539	302	253	253
   query34	919	846	508	508
   query35	731	776	646	646
   query36	828	793	732	732
   query37	132	101	81	81
   query38	1803	1742	1668	1668
   query39	809	803	740	740
   query39_1	727	736	736	736
   query40	213	118	100	100
   query41	72	71	70	70
   query42	91	93	83	83
   query43	346	375	313	313
   query44	1340	680	688	680
   query45	178	179	165	165
   query46	829	963	562	562
   query47	3020	2969	2913	2913
   query48	303	313	226	226
   query49	589	403	329	329
   query50	704	273	206	206
   query51	10492	10246	10359	10246
   query52	76	81	71	71
   query53	186	203	149	149
   query54	235	190	195	190
   query55	75	72	67	67
   query56	254	209	221	209
   query57	1670	1646	1557	1557
   query58	290	264	256	256
   query59	2070	2127	1914	1914
   query60	281	241	226	226
   query61	150	138	153	138
   query62	381	327	294	294
   query63	195	156	161	156
   query64	2755	982	827	827
   query65	3332	3281	3264	3264
   query66	1785	419	294	294
   query67	19959	20084	19889	19889
   query68	2924	1010	611	611
   query69	385	298	243	243
   query70	846	771	769	769
   query71	303	241	215	215
   query72	2752	2375	2282	2282
   query73	509	533	294	294
   query74	4661	4529	4347	4347
   query75	2343	2271	1922	1922
   query76	2316	1040	614	614
   query77	346	374	268	268
   query78	9771	9735	9214	9214
   query79	1100	847	519	519
   query80	1229	421	352	352
   query81	594	321	270	270
   query82	581	137	108	108
   query83	305	198	180	180
   query84	313	122	102	102
   query85	883	445	373	373
   query86	484	238	230	230
   query87	1986	1943	1792	1792
   query88	3761	2682	2642	2642
   query89	331	282	242	242
   query90	1863	167	158	158
   query91	158	156	127	127
   query92	96	90	89	89
   query93	1138	989	572	572
   query94	660	306	270	270
   query95	563	325	359	325
   query96	663	490	233	233
   query97	2426	2432	2372	2372
   query98	163	152	147	147
   query99	762	766	634	634
   Total cold run time: 235242 ms
   Total hot run time: 150979 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6287117595

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6266818367

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69286:
URL: https://github.com/apache/doris/pull/69286#discussion_r4314470270


##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/agg/IvmAggExtremalProcessor.java:
##########
@@ -109,48 +135,94 @@ public void appendApplyExpressions(IvmAggTarget target, IvmAggApplyContext apply
         IvmAggExpressionBuilder ctx = applyContext.expressions();
         Slot oldExtreme = applyContext.rawMvSlot(target.getValueStateColumnName());
         Expression deltaInsert = applyContext.deltaSlotValue(target, functionKind);
-        Expression deltaDel = applyContext.deltaSlotValue(target, deleteSlotRef(target));
-        Expression newCount = applyContext.buildNewHiddenCount(target);
-
+        Expression deltaDelete = applyContext.deltaSlotValue(target, deleteSlotRef(target));
+        Expression deltaInsertCount = applyContext.deltaSlotValue(target, insertCountSlotRef(target));
+        Expression deltaDeleteCount = applyContext.deltaSlotValue(target, deleteCountSlotRef(target));
+
+        Slot oldCountSlot = Objects.requireNonNull(target.getHiddenStateSlot(countStateKey),
+                "missing extreme count state for " + target);
+        Expression oldCount = ctx.zeroIfNull(applyContext.rawMvSlot(oldCountSlot.getName()));
+        Expression newTotalCount = applyContext.buildNewHiddenCount(target);
+        Expression zero = new BigIntLiteral(0L);
+
+        // Keep every predicate non-NULL: an empty insert/delete side must read as false, not
+        // as unknown, so the guard below is a plain boolean.
+        Expression insertedPresent = new Not(new IsNull(deltaInsert));
+        Expression deletedPresent = new Not(new IsNull(deltaDelete));
+        Expression insertedBetter = new And(insertedPresent, insertedIsBetter(deltaInsert, oldExtreme));
+        Expression insertedEqual = new And(insertedPresent, new EqualTo(deltaInsert, oldExtreme));
+        Expression deletedEqual = new And(deletedPresent, new EqualTo(deltaDelete, oldExtreme));
+        Expression survivingOldCount = new Subtract(
+                new Add(oldCount, new If(insertedEqual, deltaInsertCount, zero)),
+                new If(deletedEqual, deltaDeleteCount, zero));
+
+        // The old extreme survives when no delete is below it and its remaining count is
+        // positive. An inserted value that beats the old extreme also makes the merge
+        // derivable without a complete refresh.
         Expression guardCond = new Or(ImmutableList.of(
-                new EqualTo(newCount, new BigIntLiteral(0L)),
-                new IsNull(deltaDel),
+                new EqualTo(newTotalCount, zero),
                 new IsNull(oldExtreme),
-                deletedExtremeKeepsOldValueValid(deltaDel, oldExtreme, deltaInsert)
-        ));
+                new IsNull(deltaDelete),
+                new And(
+                        new And(deletedPresent,
+                                deletedExtremeIsAtLeastOld(deltaDelete, oldExtreme)),
+                        new Or(ImmutableList.of(
+                                insertedBetter,
+                                new GreaterThan(survivingOldCount, zero))))));
         Expression guard = new AssertTrue(guardCond, new StringLiteral(fallbackMessage()));
+
         Expression newExtremeRaw = new CaseWhen(
                 ImmutableList.of(
-                        new WhenClause(new EqualTo(newCount, new BigIntLiteral(0L)),
+                        new WhenClause(new EqualTo(newTotalCount, zero),
                                 new NullLiteral(oldExtreme.getDataType())),
                         new WhenClause(new IsNull(oldExtreme), deltaInsert),
-                        new WhenClause(new IsNull(deltaInsert), oldExtreme)
-                ),
-                mergeOldAndInsertedExtreme(oldExtreme, deltaInsert)
-        );
+                        new WhenClause(insertedBetter, deltaInsert)),
+                oldExtreme);
         Expression newExtremeGuarded = new If(guard, newExtremeRaw,
                 new NullLiteral(newExtremeRaw.getDataType()));
 
-        applyContext.putFinalExpression(target, target.getHiddenStateSlot(IvmAggStateKey.COUNT).getName(), newCount);
+        Expression newCountRaw = new CaseWhen(
+                ImmutableList.of(
+                        new WhenClause(new EqualTo(newTotalCount, zero), zero),
+                        new WhenClause(new IsNull(oldExtreme), deltaInsertCount),

Review Comment:
   Fixed in 87a8d552ab5. When the old extreme is NULL, the count now comes from the net insert/delete contributions at the insert-side extreme: subtract deltaDeleteCount when the delete-side extreme equals it, use deltaInsertCount when the delete extreme is farther from the aggregate extreme, and keep the complete-refresh fallback when a delete is closer or cancels every inserted row. I also added the two-refresh canceled-join regression test_ivm_minmax_count_join, which reproduced the stale MIN=1 behavior before the fix.



##########
be/test/exprs/aggregate/agg_count_at_min_max_test.cpp:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include <cstring>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "core/column/column.h"
+#include "core/column/column_string.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+#include "exprs/aggregate/aggregate_function_simple_factory.h"
+#include "exprs/aggregate/single_value_data.h"
+
+namespace doris {
+
+void register_aggregate_function_count_at_min_max(AggregateFunctionSimpleFactory& factory);
+
+namespace {
+
+AggregateFunctionPtr create_function(AggregateFunctionSimpleFactory& factory,
+                                     const std::string& name, const DataTypes& argument_types) {
+    auto function = factory.get(name, argument_types, nullptr, false, -1);
+    EXPECT_NE(function, nullptr);
+    return function;
+}
+
+void aggregate_int32(IAggregateFunction* function, Arena& arena, const IColumn& column,
+                     int64_t expected) {
+    std::unique_ptr<char[]> memory(new char[function->size_of_data()]);
+    AggregateDataPtr place = memory.get();
+    function->create(place);
+    const IColumn* columns[1] = {&column};
+    for (size_t row = 0; row < column.size(); ++row) {
+        function->add(place, columns, row, arena);
+    }
+    ColumnInt64 result;
+    function->insert_result_into(place, result);
+    EXPECT_EQ(expected, result.get_element(0));
+    function->destroy(place);
+}
+
+} // namespace
+
+TEST(CountAtMinMaxTest, Int32) {
+    AggregateFunctionSimpleFactory factory;
+    register_aggregate_function_count_at_min_max(factory);
+    DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+
+    auto column = ColumnInt32::create();
+    for (int value : {1, 2, 1, 3, 1, 2}) {
+        column->insert_value(value);
+    }
+
+    Arena arena;
+    auto count_at_min = create_function(factory, "count_at_min", data_types);
+    aggregate_int32(count_at_min.get(), arena, *column, 3);
+
+    auto count_at_max = create_function(factory, "count_at_max", data_types);
+    aggregate_int32(count_at_max.get(), arena, *column, 1);
+}
+
+TEST(CountAtMinMaxTest, EmptyInput) {
+    AggregateFunctionSimpleFactory factory;
+    register_aggregate_function_count_at_min_max(factory);
+    DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+    Arena arena;
+
+    auto column = ColumnInt32::create();
+    for (const auto& name : {"count_at_min", "count_at_max"}) {
+        auto function = create_function(factory, name, data_types);
+        aggregate_int32(function.get(), arena, *column, 0);
+    }
+}
+
+TEST(CountAtMinMaxTest, Merge) {
+    AggregateFunctionSimpleFactory factory;
+    register_aggregate_function_count_at_min_max(factory);
+    DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+    Arena arena;
+
+    auto first = ColumnInt32::create();
+    auto second = ColumnInt32::create();
+    for (int value : {5, 1, 5}) {
+        first->insert_value(value);
+    }
+    for (int value : {5, 1, 9}) {
+        second->insert_value(value);
+    }
+
+    auto function = create_function(factory, "count_at_min", data_types);
+    std::unique_ptr<char[]> memory(new char[function->size_of_data() * 2]);
+    AggregateDataPtr lhs = memory.get();
+    AggregateDataPtr rhs = memory.get() + function->size_of_data();
+    function->create(lhs);
+    function->create(rhs);
+
+    const IColumn* first_columns[1] = {first.get()};
+    const IColumn* second_columns[1] = {second.get()};
+    for (size_t row = 0; row < first->size(); ++row) {
+        function->add(lhs, first_columns, row, arena);
+    }
+    for (size_t row = 0; row < second->size(); ++row) {
+        function->add(rhs, second_columns, row, arena);
+    }
+    function->merge(lhs, rhs, arena);
+
+    ColumnInt64 result;
+    function->insert_result_into(lhs, result);
+    EXPECT_EQ(5, result.get_element(0));

Review Comment:
   Fixed in 87a8d552ab5. The merge fixture now expects 2: each partial has minimum 1 with count 1, so merging equal extrema returns 2.



##########
regression-test/suites/mtmv_p0/ivm/test_ivm_minmax_count.groovy:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.awaitility.Awaitility
+import static java.util.concurrent.TimeUnit.SECONDS
+
+suite("test_ivm_minmax_count") {
+    // The base-table queries below assert the same rows as the MV, so transparent rewrite must be off.
+    sql """set enable_materialized_view_rewrite = false;"""
+    sql """set enable_dml_materialized_view_rewrite = false;"""
+    sql """drop materialized view if exists ivm_mm_count_mv"""
+    sql """drop table if exists ivm_mm_count_t"""
+
+    sql """
+        CREATE TABLE ivm_mm_count_t (
+            id BIGINT NOT NULL,
+            category VARCHAR(16) NOT NULL,
+            score BIGINT
+        )
+        UNIQUE KEY(id)
+        DISTRIBUTED BY HASH(id) BUCKETS 1
+        PROPERTIES (
+            "replication_num" = "1",
+            "enable_unique_key_merge_on_write" = "true",
+            "binlog.enable" = "true",
+            "binlog.format" = "ROW",
+            "binlog.need_historical_value" = "true"
+        )
+    """
+
+    // category a: min=10 twice, max=30 once
+    // category b: min=5 three times, max=5
+    sql """INSERT INTO ivm_mm_count_t VALUES
+        (1, 'a', 10), (2, 'a', 10), (3, 'a', 30),
+        (4, 'b', 5),  (5, 'b', 5),  (6, 'b', 5)"""
+
+    sql """
+        CREATE MATERIALIZED VIEW ivm_mm_count_mv
+        BUILD DEFERRED REFRESH INCREMENTAL FALLBACK ON MANUAL
+        KEY(category)
+        DISTRIBUTED BY HASH(category) BUCKETS 1
+        PROPERTIES ("replication_num" = "1")
+        AS SELECT category, MIN(score) AS min_score, MAX(score) AS max_score, COUNT(*) AS row_count
+        FROM ivm_mm_count_t
+        GROUP BY category
+    """
+
+    def assertNoFallback = { task ->
+        def reason = task.IvmFallbackReason
+        assertTrue(reason == null || reason.toString() == "\\N",
+                "unexpected IVM fallback: ${reason}")
+    }
+
+    def latestTask = {
+        def taskResult
+        Awaitility.await().atMost(300, SECONDS).pollInterval(2, SECONDS).until({
+            taskResult = sql_return_maparray("""
+                SELECT Status, RefreshMode, IvmFallbackReason, ErrorMsg
+                FROM tasks('type'='mv')
+                WHERE MvDatabaseName = '${context.dbName}'
+                  AND MvName = 'ivm_mm_count_mv'
+                ORDER BY CreateTime DESC, TaskId DESC LIMIT 1
+            """)
+            return !taskResult.isEmpty()
+                    && taskResult[0].Status.toString() != 'PENDING'
+                    && taskResult[0].Status.toString() != 'RUNNING'

Review Comment:
   Fixed in 87a8d552ab5. Both MIN/MAX count suites now remember the previous TaskId and only accept a terminal row whose TaskId differs, so the running-list-to-history gap can no longer return the preceding refresh. The new join suite uses the same wait.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6268543523

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6271983887

   # FE Regression Coverage Report
   Increment line coverage `  79.37% (100/126)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6272882368

   # FE Regression Coverage Report
   Increment line coverage `  79.37% (100/126)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282048615

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282057620

   Codex automated review failed and did not complete.
   
   Error: Review context preparation failed before Codex ran; inspect the 'Prepare authoritative PR context and required AGENTS guides' step.
   Workflow run: https://github.com/apache/doris/actions/runs/39358482180
   
   Please inspect the workflow logs and rerun the review after the underlying issue is resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281321162

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282447718

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 150970 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 008d08b586b9e981e19d07e8e5ca01d63f244045, data reload: false
   
   query5	4294	586	446	446
   query6	428	206	191	191
   query7	4819	440	236	236
   query8	326	161	146	146
   query9	8760	3159	3171	3159
   query10	485	320	282	282
   query11	6047	3838	3594	3594
   query12	146	92	89	89
   query13	1254	450	338	338
   query14	6078	3930	3620	3620
   query14_1	3446	3374	3424	3374
   query15	211	201	179	179
   query16	972	468	468	468
   query17	876	666	545	545
   query18	2424	443	320	320
   query19	196	167	136	136
   query20	88	85	89	85
   query21	215	134	108	108
   query22	12999	12999	12822	12822
   query23	13422	12887	12531	12531
   query23_1	12616	12474	12507	12474
   query24	7388	840	463	463
   query24_1	481	495	476	476
   query25	550	420	365	365
   query26	1252	269	143	143
   query27	2760	439	272	272
   query28	4638	1943	1948	1943
   query29	1575	610	490	490
   query30	320	227	188	188
   query31	1011	935	822	822
   query32	162	92	94	92
   query33	537	316	245	245
   query34	944	840	506	506
   query35	752	783	647	647
   query36	827	785	730	730
   query37	134	101	87	87
   query38	1797	1741	1686	1686
   query39	795	830	741	741
   query39_1	729	725	732	725
   query40	223	116	99	99
   query41	75	68	71	68
   query42	88	89	92	89
   query43	354	362	317	317
   query44	1379	687	697	687
   query45	183	177	160	160
   query46	845	967	566	566
   query47	2961	3058	2816	2816
   query48	298	318	213	213
   query49	570	399	297	297
   query50	700	262	206	206
   query51	10191	10439	10060	10060
   query52	75	81	68	68
   query53	191	204	150	150
   query54	253	225	183	183
   query55	72	71	63	63
   query56	229	208	207	207
   query57	1638	1618	1565	1565
   query58	292	265	252	252
   query59	2104	2145	1921	1921
   query60	294	237	224	224
   query61	145	147	145	145
   query62	385	333	287	287
   query63	199	163	162	162
   query64	2744	944	744	744
   query65	3299	3285	3284	3284
   query66	1785	421	306	306
   query67	20355	20224	20050	20050
   query68	2973	1007	599	599
   query69	387	281	236	236
   query70	894	775	760	760
   query71	304	232	214	214
   query72	2709	2558	2543	2543
   query73	518	543	301	301
   query74	4646	4533	4373	4373
   query75	2341	2295	1909	1909
   query76	2320	1029	622	622
   query77	352	366	275	275
   query78	9940	9806	9104	9104
   query79	953	899	516	516
   query80	830	420	335	335
   query81	580	319	271	271
   query82	574	131	111	111
   query83	305	194	185	185
   query84	307	119	104	104
   query85	897	436	368	368
   query86	360	244	230	230
   query87	1987	1971	1801	1801
   query88	3686	2680	2650	2650
   query89	343	284	237	237
   query90	1808	171	171	171
   query91	174	155	128	128
   query92	94	87	86	86
   query93	1005	991	570	570
   query94	514	306	271	271
   query95	581	386	308	308
   query96	680	495	230	230
   query97	2457	2436	2337	2337
   query98	165	152	146	146
   query99	765	760	660	660
   Total cold run time: 234809 ms
   Total hot run time: 150970 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org