You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yujun777 (via GitHub)" <gi...@apache.org> on 2026/10/19 12:07:06 UTC
[PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
yujun777 opened a new pull request, #69286:
URL: https://github.com/apache/doris/pull/69286
### What problem does this PR solve?
Incremental refresh of a MIN/MAX materialized view currently falls back to a
complete refresh whenever a deleted row could be the current extreme. Deleting
one of several rows that share the minimum value is a common case, yet the MV
does not remember how many rows hold that extreme, so the refresh cannot tell
that the extreme survives.
This PR adds `COUNT_AT_MIN` / `COUNT_AT_MAX` and stores that count as a hidden
IVM state column. The delta plan now emits the count at the insert-side and
delete-side extreme. Apply keeps the old extreme while its surviving count is
positive; only deleting the last row at the extreme still falls back to a
complete refresh.
Trace issue: https://github.com/apache/doris/issues/65418
### What changed?
- Add public `COUNT_AT_MIN(expr)` / `COUNT_AT_MAX(expr)` aggregate functions.
They return the number of input rows equal to the group's minimum/maximum,
ignore NULLs, return 0 for empty or all-NULL input, and reject DISTINCT.
They are equivalent to Oracle's
`COUNT(*) KEEP (DENSE_RANK FIRST/LAST ORDER BY expr)`.
- Add `MIN_COUNT` / `MAX_COUNT` IVM hidden state keys and hidden columns named
in the existing `__DORIS_IVM_AGG_{ordinal}_{STATE}_COL__` format.
- Emit insert/delete extreme counts in the delta aggregate.
- Use the surviving extreme count in the apply guard and update the hidden
count state.
- Add BE/FE unit coverage and a regression suite.
### Testing
- FE unit tests: `IvmAggMinProcessorTest`, `IvmAggMaxProcessorTest`,
`IvmAggDeltaHandlerTest`, `IvmNormalizeUniqKeysMTMVTest`.
- Regression: `test_ivm_minmax_count`, plus existing
`test_ivm_minmax_runtime_fallback`, `test_ivm_agg_1`, `test_ivm_agg_2`, and
`test_ivm_agg_shared_columns`.
- SQL verification of duplicate extremes, all-NULL input, and DISTINCT
rejection.
- FE build/checkstyle, BE build, clang-format/check-format, and build hygiene
pass.
- A BE unit test for the new aggregate is included. The local BE unit-test
target could not link in this environment because the toolchain already
provides glibc-compatibility symbols; the main BE build succeeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6268544096
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6284559334
# BE Regression && UT Coverage Report
Increment line coverage `92.05% (81/88)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.23% (35815/46376) |
| Line Coverage | 62.47% (404199/647048) |
| Region Coverage | 58.95% (341456/579251) |
| Branch Coverage | 59.86% (157751/263526) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 merged PR #69286:
URL: https://github.com/apache/doris/pull/69286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282323915
<details>
<summary>TPC-H: <b>Total hot run time: 28433 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 008d08b586b9e981e19d07e8e5ca01d63f244045, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17611 5230 5036 5036
q2 2103 313 276 276
q3 10223 1393 844 844
q4 4681 467 356 356
q5 7448 832 527 527
q6 180 187 150 150
q7 750 815 638 638
q8 9255 1368 1052 1052
q9 5373 4221 4186 4186
q10 6857 1289 1030 1030
q11 422 247 232 232
q12 631 395 285 285
q13 18095 3689 3026 3026
q14 286 273 246 246
q15 q16 741 704 644 644
q17 1224 752 582 582
q18 7062 6182 6091 6091
q19 1255 959 605 605
q20 384 330 226 226
q21 5330 2153 2268 2153
q22 362 307 248 248
Total cold run time: 100273 ms
Total hot run time: 28433 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5615 5627 5603 5603
q2 702 547 524 524
q3 5187 5360 4914 4914
q4 1999 2147 1545 1545
q5 5280 5136 5123 5123
q6 235 181 146 146
q7 2069 1826 1711 1711
q8 2844 2456 2453 2453
q9 7508 7527 7538 7527
q10 3651 3570 3124 3124
q11 578 433 378 378
q12 663 670 504 504
q13 3167 3541 3058 3058
q14 290 293 271 271
q15 q16 670 699 623 623
q17 7723 6982 6924 6924
q18 12974 12172 12908 12172
q19 944 831 853 831
q20 2125 2164 1914 1914
q21 5632 4646 4643 4643
q22 491 446 441 441
Total cold run time: 70347 ms
Total hot run time: 64429 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69286:
URL: https://github.com/apache/doris/pull/69286#discussion_r4320711849
##########
be/src/exprs/aggregate/aggregate_function_count_at_min_max.cpp:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/aggregate/aggregate_function_count_at_min_max.h"
+
+#include "common/exception.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/define_primitive_type.h"
+#include "core/data_type/primitive_type.h"
+#include "exprs/aggregate/aggregate_function_simple_factory.h"
+#include "exprs/aggregate/factory_helpers.h"
+#include "exprs/aggregate/helpers.h"
+
+namespace doris {
+
+namespace {
+
+template <template <typename> class ExtremeData>
+AggregateFunctionPtr create_aggregate_function_count_at_extreme(const String& name,
+ const DataTypes& argument_types,
+ const DataTypePtr& result_type,
+ const bool result_is_nullable,
+ const AggregateFunctionAttr& attr) {
+ assert_arity_range(name, argument_types, 1, 1);
+ switch (argument_types[0]->get_primitive_type()) {
+ case PrimitiveType::TYPE_STRING:
+ case PrimitiveType::TYPE_CHAR:
+ case PrimitiveType::TYPE_VARCHAR:
+ case PrimitiveType::TYPE_JSONB:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataString>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_DATE:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATE>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_DATETIME:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATETIME>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_DATEV2:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATEV2>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_DATETIMEV2:
+ return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+ ExtremeData<SingleValueDataFixed<TYPE_DATETIMEV2>>>>(argument_types,
+ result_is_nullable, attr);
+ case PrimitiveType::TYPE_TIMESTAMP_NS:
+ return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+ ExtremeData<SingleValueDataFixed<TYPE_TIMESTAMP_NS>>>>(argument_types,
+ result_is_nullable, attr);
+ case PrimitiveType::TYPE_TIMESTAMPTZ:
+ return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+ ExtremeData<SingleValueDataFixed<TYPE_TIMESTAMPTZ>>>>(argument_types,
+ result_is_nullable, attr);
+ case PrimitiveType::TYPE_TIMEV2:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_TIMEV2>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_IPV4:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_IPV4>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_IPV6:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_IPV6>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_UUID:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_UUID>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_BOOLEAN:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_BOOLEAN>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_TINYINT:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_TINYINT>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_SMALLINT:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_SMALLINT>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_INT:
+ return creator_without_type::create_unary_arguments<
Review Comment:
[P2] Preserve zero results in the foreach variants
For an `ARRAY<INT>` column `a` containing `[1,NULL]` and `[2,NULL]`, both `count_at_min_foreach(a)` and `count_at_max_foreach(a)` produce `[1,NULL]` instead of `[1,0]`. The FE exposes these combinators automatically, and `foreachv2` requests `result_is_nullable=true` for its nested function. Forwarding that flag through `create_unary_arguments` installs a nullable-result adapter for the nullable array elements; an all-NULL position then emits NULL without calling the count state's zero-result insertion. Both old and V2 nullable adapters take this path. Use the nonnullable-result unary creator in all type branches (or otherwise preserve the nested aggregate's nonnullable semantics), and add an all-NULL-element regression for both variants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281835541
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6283129100
# FE UT Coverage Report
Increment line coverage ` 71.43% (90/126)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6266817858
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6271286998
<details>
<summary>ClickBench: <b>Total hot run time: 24.32 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 87a8d552ab57c552322af4ce4a77ca290d6829ac, data reload: false
query1 0.01 0.01 0.00
query2 0.09 0.05 0.05
query3 0.26 0.09 0.08
query4 1.60 0.10 0.10
query5 0.23 0.25 0.23
query6 1.17 0.69 0.66
query7 0.04 0.01 0.01
query8 0.05 0.04 0.03
query9 0.38 0.33 0.36
query10 0.60 0.61 0.57
query11 0.21 0.15 0.14
query12 0.19 0.15 0.15
query13 0.49 0.48 0.48
query14 1.03 1.02 1.01
query15 0.63 0.60 0.59
query16 0.34 0.35 0.35
query17 1.14 1.18 1.15
query18 0.21 0.19 0.19
query19 2.09 2.05 2.00
query20 0.02 0.01 0.02
query21 15.48 0.20 0.14
query22 4.80 0.06 0.05
query23 16.25 0.30 0.11
query24 3.11 0.62 0.49
query25 0.09 0.05 0.06
query26 1.59 0.22 0.17
query27 0.05 0.04 0.05
query28 3.48 0.77 0.35
query29 12.50 4.10 3.22
query30 0.28 0.14 0.10
query31 2.77 0.55 0.32
query32 3.22 0.60 0.51
query33 3.22 3.15 3.22
query34 15.51 4.11 3.39
query35 3.34 3.34 3.35
query36 0.57 0.44 0.42
query37 0.10 0.06 0.07
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.17 0.14 0.14
query41 0.09 0.03 0.03
query42 0.03 0.02 0.03
query43 0.03 0.03 0.04
Total cold run time: 97.55 s
Total hot run time: 24.32 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6271201127
<details>
<summary>TPC-DS: <b>Total hot run time: 150262 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 87a8d552ab57c552322af4ce4a77ca290d6829ac, data reload: false
query5 4299 579 433 433
query6 426 223 191 191
query7 4809 440 228 228
query8 351 170 151 151
query9 8762 3035 3037 3035
query10 471 303 248 248
query11 6001 3843 3569 3569
query12 143 95 88 88
query13 1255 466 327 327
query14 6155 3908 3621 3621
query14_1 3360 3399 3376 3376
query15 209 197 180 180
query16 933 424 404 404
query17 885 701 528 528
query18 2440 436 312 312
query19 212 173 143 143
query20 94 82 82 82
query21 212 134 109 109
query22 12979 13034 12783 12783
query23 13424 12911 12441 12441
query23_1 12607 12502 12517 12502
query24 7383 809 459 459
query24_1 469 472 466 466
query25 531 415 368 368
query26 1262 253 145 145
query27 2770 438 269 269
query28 4638 1910 1935 1910
query29 1553 603 462 462
query30 298 219 185 185
query31 1012 929 829 829
query32 139 94 95 94
query33 543 313 247 247
query34 886 853 495 495
query35 728 796 651 651
query36 821 814 729 729
query37 126 98 84 84
query38 1745 1749 1714 1714
query39 791 800 738 738
query39_1 731 724 744 724
query40 215 119 97 97
query41 73 68 67 67
query42 87 86 82 82
query43 338 362 316 316
query44 1293 677 683 677
query45 179 173 168 168
query46 822 964 563 563
query47 2801 3028 2852 2852
query48 296 307 250 250
query49 569 420 288 288
query50 657 262 211 211
query51 10437 10261 10590 10261
query52 76 79 75 75
query53 182 214 150 150
query54 231 195 177 177
query55 69 75 61 61
query56 237 212 217 212
query57 1640 1598 1554 1554
query58 292 256 256 256
query59 2148 2196 1928 1928
query60 288 237 210 210
query61 163 147 142 142
query62 384 338 276 276
query63 192 160 155 155
query64 2718 974 766 766
query65 3289 3241 3251 3241
query66 1775 413 297 297
query67 20421 20138 19972 19972
query68 3026 992 591 591
query69 383 274 244 244
query70 860 768 735 735
query71 283 237 210 210
query72 2705 2572 2267 2267
query73 508 525 287 287
query74 4586 4462 4326 4326
query75 2312 2293 1920 1920
query76 2299 994 589 589
query77 341 364 272 272
query78 9692 9664 9146 9146
query79 984 841 516 516
query80 1254 427 333 333
query81 587 327 264 264
query82 580 142 104 104
query83 306 200 178 178
query84 308 115 101 101
query85 859 414 371 371
query86 402 239 229 229
query87 2016 1939 1808 1808
query88 3612 2654 2628 2628
query89 334 284 241 241
query90 1933 174 162 162
query91 163 149 124 124
query92 95 86 91 86
query93 1012 992 570 570
query94 645 293 250 250
query95 569 405 297 297
query96 659 478 228 228
query97 2452 2452 2377 2377
query98 166 151 148 148
query99 762 768 648 648
Total cold run time: 234685 ms
Total hot run time: 150262 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6287116624
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281385633
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282048300
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6269868681
<details>
<summary>TPC-H: <b>Total hot run time: 28306 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c5d12fabf6c91e627068dd92592f5b5fd2aa1e62, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17642 5179 5021 5021
q2 2111 288 251 251
q3 10281 1348 835 835
q4 4679 453 335 335
q5 7427 800 539 539
q6 181 178 149 149
q7 743 803 595 595
q8 9252 1262 1056 1056
q9 5440 4203 4204 4203
q10 6840 1282 1041 1041
q11 438 250 230 230
q12 639 399 286 286
q13 18084 3584 2973 2973
q14 272 273 242 242
q15 q16 708 709 658 658
q17 1193 750 584 584
q18 6984 6176 6115 6115
q19 1088 966 601 601
q20 383 333 226 226
q21 5085 2110 2311 2110
q22 341 293 256 256
Total cold run time: 99811 ms
Total hot run time: 28306 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5625 5635 5682 5635
q2 660 590 581 581
q3 5354 5504 5026 5026
q4 2052 2162 1551 1551
q5 5337 5122 5609 5122
q6 248 195 156 156
q7 2304 2036 1863 1863
q8 2974 2720 2694 2694
q9 8049 7904 7927 7904
q10 3738 3726 3468 3468
q11 609 453 411 411
q12 746 724 533 533
q13 3495 3794 3300 3300
q14 323 348 308 308
q15 q16 706 705 658 658
q17 8134 7423 6831 6831
q18 13017 12158 12830 12158
q19 922 790 787 787
q20 2150 2142 1876 1876
q21 5596 4572 4627 4572
q22 476 441 390 390
Total cold run time: 72515 ms
Total hot run time: 65824 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6266815938
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6272583103
# FE Regression Coverage Report
Increment line coverage ` 79.37% (100/126)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6272792414
# FE Regression Coverage Report
Increment line coverage ` 79.37% (100/126)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282505041
<details>
<summary>ClickBench: <b>Total hot run time: 24.36 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 008d08b586b9e981e19d07e8e5ca01d63f244045, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.05
query3 0.26 0.10 0.09
query4 1.60 0.10 0.09
query5 0.23 0.26 0.24
query6 1.16 0.71 0.68
query7 0.04 0.00 0.00
query8 0.05 0.04 0.03
query9 0.41 0.34 0.35
query10 0.57 0.56 0.56
query11 0.20 0.14 0.15
query12 0.18 0.15 0.15
query13 0.50 0.49 0.49
query14 1.04 1.00 1.02
query15 0.64 0.63 0.61
query16 0.31 0.35 0.39
query17 1.13 1.12 1.13
query18 0.19 0.19 0.19
query19 2.06 1.99 1.99
query20 0.02 0.01 0.01
query21 15.49 0.19 0.14
query22 4.97 0.06 0.05
query23 16.24 0.32 0.13
query24 3.07 0.49 0.44
query25 0.08 0.06 0.05
query26 0.99 0.21 0.16
query27 0.03 0.03 0.04
query28 3.47 0.77 0.37
query29 12.49 4.15 3.31
query30 0.29 0.12 0.10
query31 2.77 0.56 0.31
query32 3.22 0.60 0.50
query33 3.15 3.32 3.18
query34 15.46 4.00 3.40
query35 3.34 3.29 3.36
query36 0.54 0.48 0.41
query37 0.09 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.14 0.13
query41 0.08 0.03 0.03
query42 0.03 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 96.79 s
Total hot run time: 24.36 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281835214
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6271027430
<details>
<summary>TPC-H: <b>Total hot run time: 28165 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 87a8d552ab57c552322af4ce4a77ca290d6829ac, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17587 5103 5005 5005
q2 2109 304 245 245
q3 10258 1422 822 822
q4 4693 459 338 338
q5 7424 790 532 532
q6 179 188 154 154
q7 731 784 585 585
q8 9241 1343 1039 1039
q9 5431 4172 4162 4162
q10 6813 1277 1010 1010
q11 430 244 221 221
q12 646 394 277 277
q13 18105 3539 2966 2966
q14 272 270 245 245
q15 q16 701 719 654 654
q17 1182 751 578 578
q18 6965 6152 6115 6115
q19 1113 949 606 606
q20 387 327 226 226
q21 5009 2132 2312 2132
q22 342 295 253 253
Total cold run time: 99618 ms
Total hot run time: 28165 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5593 5576 5566 5566
q2 674 574 548 548
q3 5223 5359 4949 4949
q4 1980 2151 1559 1559
q5 5251 5158 5425 5158
q6 250 195 146 146
q7 2335 2071 1855 1855
q8 3033 2669 2717 2669
q9 8074 8053 7841 7841
q10 3745 3653 3238 3238
q11 611 484 470 470
q12 682 708 522 522
q13 3476 3915 3266 3266
q14 331 340 301 301
q15 q16 730 706 650 650
q17 8257 7185 6855 6855
q18 12947 12227 12836 12227
q19 898 798 775 775
q20 2129 2177 1909 1909
q21 5546 4554 4645 4554
q22 481 463 399 399
Total cold run time: 72246 ms
Total hot run time: 65457 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6286939023
# FE Regression Coverage Report
Increment line coverage ` 60.34% (108/179)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_008d08b586b9e981e19d07e8e5ca01d63f244045_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6270074647
<details>
<summary>TPC-DS: <b>Total hot run time: 150451 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c5d12fabf6c91e627068dd92592f5b5fd2aa1e62, data reload: false
query5 4325 580 436 436
query6 438 205 194 194
query7 4815 455 227 227
query8 311 168 156 156
query9 8790 3124 3113 3113
query10 486 304 252 252
query11 5961 3838 3573 3573
query12 146 92 87 87
query13 1259 440 322 322
query14 6141 3967 3650 3650
query14_1 3443 3429 3466 3429
query15 208 194 178 178
query16 966 441 409 409
query17 855 660 553 553
query18 2415 412 315 315
query19 190 171 132 132
query20 88 94 87 87
query21 206 126 110 110
query22 12984 12936 12852 12852
query23 13410 12775 12395 12395
query23_1 12461 12538 12431 12431
query24 7380 818 461 461
query24_1 480 476 462 462
query25 535 424 365 365
query26 1256 254 142 142
query27 2760 438 272 272
query28 4653 1934 1928 1928
query29 1565 604 477 477
query30 302 224 183 183
query31 1016 935 801 801
query32 151 95 94 94
query33 531 304 248 248
query34 954 846 495 495
query35 721 773 622 622
query36 805 820 721 721
query37 120 92 79 79
query38 1747 1701 1660 1660
query39 786 798 744 744
query39_1 730 727 740 727
query40 213 123 92 92
query41 68 63 62 62
query42 84 84 81 81
query43 344 371 311 311
query44 1277 673 679 673
query45 176 173 169 169
query46 841 941 559 559
query47 3012 3051 2889 2889
query48 296 302 214 214
query49 578 389 276 276
query50 667 267 208 208
query51 10548 10237 10160 10160
query52 79 78 70 70
query53 181 203 153 153
query54 232 189 176 176
query55 75 71 64 64
query56 294 230 208 208
query57 1635 1627 1558 1558
query58 287 255 251 251
query59 2169 2208 1965 1965
query60 295 243 227 227
query61 146 149 152 149
query62 387 328 292 292
query63 187 158 160 158
query64 2735 917 769 769
query65 3304 3241 3268 3241
query66 1771 437 292 292
query67 20391 19958 20254 19958
query68 2975 1005 603 603
query69 377 291 250 250
query70 821 788 783 783
query71 288 213 203 203
query72 2752 2579 2343 2343
query73 525 518 298 298
query74 4603 4500 4375 4375
query75 2337 2285 1900 1900
query76 2321 1001 619 619
query77 356 376 273 273
query78 9861 9698 9133 9133
query79 997 838 527 527
query80 690 448 361 361
query81 555 313 272 272
query82 575 139 107 107
query83 295 203 182 182
query84 311 128 100 100
query85 948 415 362 362
query86 336 236 234 234
query87 2005 1977 1804 1804
query88 3612 2656 2618 2618
query89 331 280 242 242
query90 1831 166 174 166
query91 165 149 128 128
query92 95 89 81 81
query93 995 979 577 577
query94 478 312 261 261
query95 582 385 301 301
query96 671 478 228 228
query97 2426 2455 2336 2336
query98 170 150 146 146
query99 777 760 651 651
Total cold run time: 234517 ms
Total hot run time: 150451 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6270177849
<details>
<summary>ClickBench: <b>Total hot run time: 24.09 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c5d12fabf6c91e627068dd92592f5b5fd2aa1e62, data reload: false
query1 0.00 0.00 0.01
query2 0.10 0.05 0.05
query3 0.26 0.09 0.08
query4 1.61 0.10 0.10
query5 0.23 0.24 0.24
query6 1.18 0.70 0.67
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.41 0.36 0.36
query10 0.60 0.57 0.59
query11 0.20 0.14 0.14
query12 0.19 0.14 0.15
query13 0.48 0.49 0.50
query14 1.04 1.02 1.00
query15 0.62 0.59 0.60
query16 0.36 0.34 0.30
query17 1.15 1.09 1.14
query18 0.20 0.20 0.18
query19 2.03 2.04 1.98
query20 0.02 0.01 0.02
query21 15.50 0.19 0.14
query22 4.83 0.06 0.05
query23 16.38 0.32 0.12
query24 2.86 0.48 0.41
query25 0.10 0.06 0.04
query26 1.22 0.21 0.15
query27 0.04 0.05 0.03
query28 3.29 0.82 0.36
query29 12.48 4.09 3.24
query30 0.27 0.14 0.10
query31 2.77 0.56 0.32
query32 3.24 0.58 0.51
query33 3.28 3.13 3.32
query34 15.43 3.93 3.36
query35 3.33 3.32 3.34
query36 0.57 0.45 0.43
query37 0.09 0.07 0.06
query38 0.05 0.03 0.04
query39 0.04 0.03 0.03
query40 0.17 0.14 0.14
query41 0.08 0.03 0.02
query42 0.03 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 96.86 s
Total hot run time: 24.09 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69286:
URL: https://github.com/apache/doris/pull/69286#discussion_r4320918485
##########
be/src/exprs/aggregate/aggregate_function_count_at_min_max.cpp:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/aggregate/aggregate_function_count_at_min_max.h"
+
+#include "common/exception.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/define_primitive_type.h"
+#include "core/data_type/primitive_type.h"
+#include "exprs/aggregate/aggregate_function_simple_factory.h"
+#include "exprs/aggregate/factory_helpers.h"
+#include "exprs/aggregate/helpers.h"
+
+namespace doris {
+
+namespace {
+
+template <template <typename> class ExtremeData>
+AggregateFunctionPtr create_aggregate_function_count_at_extreme(const String& name,
+ const DataTypes& argument_types,
+ const DataTypePtr& result_type,
+ const bool result_is_nullable,
+ const AggregateFunctionAttr& attr) {
+ assert_arity_range(name, argument_types, 1, 1);
+ switch (argument_types[0]->get_primitive_type()) {
+ case PrimitiveType::TYPE_STRING:
+ case PrimitiveType::TYPE_CHAR:
+ case PrimitiveType::TYPE_VARCHAR:
+ case PrimitiveType::TYPE_JSONB:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataString>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_DATE:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATE>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_DATETIME:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATETIME>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_DATEV2:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_DATEV2>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_DATETIMEV2:
+ return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+ ExtremeData<SingleValueDataFixed<TYPE_DATETIMEV2>>>>(argument_types,
+ result_is_nullable, attr);
+ case PrimitiveType::TYPE_TIMESTAMP_NS:
+ return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+ ExtremeData<SingleValueDataFixed<TYPE_TIMESTAMP_NS>>>>(argument_types,
+ result_is_nullable, attr);
+ case PrimitiveType::TYPE_TIMESTAMPTZ:
+ return creator_without_type::create_unary_arguments<AggregateFunctionsCountAtExtreme<
+ ExtremeData<SingleValueDataFixed<TYPE_TIMESTAMPTZ>>>>(argument_types,
+ result_is_nullable, attr);
+ case PrimitiveType::TYPE_TIMEV2:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_TIMEV2>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_IPV4:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_IPV4>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_IPV6:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_IPV6>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_UUID:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_UUID>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_BOOLEAN:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_BOOLEAN>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_TINYINT:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_TINYINT>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_SMALLINT:
+ return creator_without_type::create_unary_arguments<
+ AggregateFunctionsCountAtExtreme<ExtremeData<SingleValueDataFixed<TYPE_SMALLINT>>>>(
+ argument_types, result_is_nullable, attr);
+ case PrimitiveType::TYPE_INT:
+ return creator_without_type::create_unary_arguments<
Review Comment:
Fixed in 008d08b586b. The nested count_at_min/count_at_max now uses create_unary_arguments_return_not_nullable, so the foreach wrapper keeps the non-nullable zero result instead of emitting NULL. I added an all-NULL array position regression for both count_at_min_foreach and count_at_max_foreach, under the default V2 and the old nullable adapter; both now return [1,0].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282033674
<details>
<summary>ClickBench: <b>Total hot run time: 24.13 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fe394a7d126524423d42188559245ad083f7f3c6, data reload: false
query1 0.00 0.00 0.00
query2 0.09 0.04 0.05
query3 0.26 0.09 0.09
query4 1.61 0.11 0.11
query5 0.25 0.23 0.25
query6 1.18 0.70 0.68
query7 0.05 0.01 0.01
query8 0.06 0.04 0.04
query9 0.39 0.35 0.33
query10 0.56 0.56 0.59
query11 0.20 0.15 0.14
query12 0.18 0.15 0.15
query13 0.50 0.48 0.50
query14 1.05 1.01 1.02
query15 0.62 0.60 0.60
query16 0.36 0.35 0.32
query17 1.14 1.15 1.11
query18 0.20 0.20 0.20
query19 2.12 1.95 1.99
query20 0.02 0.01 0.01
query21 15.49 0.18 0.13
query22 4.91 0.06 0.05
query23 16.29 0.32 0.12
query24 3.15 0.55 0.33
query25 0.11 0.05 0.06
query26 0.92 0.23 0.15
query27 0.05 0.04 0.04
query28 2.51 0.73 0.36
query29 12.50 4.09 3.25
query30 0.28 0.13 0.12
query31 2.77 0.55 0.32
query32 3.23 0.59 0.50
query33 3.12 3.30 3.18
query34 15.44 3.99 3.38
query35 3.33 3.34 3.32
query36 0.54 0.46 0.42
query37 0.10 0.07 0.07
query38 0.04 0.03 0.04
query39 0.04 0.03 0.03
query40 0.18 0.14 0.14
query41 0.09 0.02 0.03
query42 0.04 0.03 0.04
query43 0.04 0.03 0.04
Total cold run time: 96.01 s
Total hot run time: 24.13 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69286:
URL: https://github.com/apache/doris/pull/69286#discussion_r4313715856
##########
be/test/exprs/aggregate/agg_count_at_min_max_test.cpp:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include <cstring>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "core/column/column.h"
+#include "core/column/column_string.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+#include "exprs/aggregate/aggregate_function_simple_factory.h"
+#include "exprs/aggregate/single_value_data.h"
+
+namespace doris {
+
+void register_aggregate_function_count_at_min_max(AggregateFunctionSimpleFactory& factory);
+
+namespace {
+
+AggregateFunctionPtr create_function(AggregateFunctionSimpleFactory& factory,
+ const std::string& name, const DataTypes& argument_types) {
+ auto function = factory.get(name, argument_types, nullptr, false, -1);
+ EXPECT_NE(function, nullptr);
+ return function;
+}
+
+void aggregate_int32(IAggregateFunction* function, Arena& arena, const IColumn& column,
+ int64_t expected) {
+ std::unique_ptr<char[]> memory(new char[function->size_of_data()]);
+ AggregateDataPtr place = memory.get();
+ function->create(place);
+ const IColumn* columns[1] = {&column};
+ for (size_t row = 0; row < column.size(); ++row) {
+ function->add(place, columns, row, arena);
+ }
+ ColumnInt64 result;
+ function->insert_result_into(place, result);
+ EXPECT_EQ(expected, result.get_element(0));
+ function->destroy(place);
+}
+
+} // namespace
+
+TEST(CountAtMinMaxTest, Int32) {
+ AggregateFunctionSimpleFactory factory;
+ register_aggregate_function_count_at_min_max(factory);
+ DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+
+ auto column = ColumnInt32::create();
+ for (int value : {1, 2, 1, 3, 1, 2}) {
+ column->insert_value(value);
+ }
+
+ Arena arena;
+ auto count_at_min = create_function(factory, "count_at_min", data_types);
+ aggregate_int32(count_at_min.get(), arena, *column, 3);
+
+ auto count_at_max = create_function(factory, "count_at_max", data_types);
+ aggregate_int32(count_at_max.get(), arena, *column, 1);
+}
+
+TEST(CountAtMinMaxTest, EmptyInput) {
+ AggregateFunctionSimpleFactory factory;
+ register_aggregate_function_count_at_min_max(factory);
+ DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+ Arena arena;
+
+ auto column = ColumnInt32::create();
+ for (const auto& name : {"count_at_min", "count_at_max"}) {
+ auto function = create_function(factory, name, data_types);
+ aggregate_int32(function.get(), arena, *column, 0);
+ }
+}
+
+TEST(CountAtMinMaxTest, Merge) {
+ AggregateFunctionSimpleFactory factory;
+ register_aggregate_function_count_at_min_max(factory);
+ DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+ Arena arena;
+
+ auto first = ColumnInt32::create();
+ auto second = ColumnInt32::create();
+ for (int value : {5, 1, 5}) {
+ first->insert_value(value);
+ }
+ for (int value : {5, 1, 9}) {
+ second->insert_value(value);
+ }
+
+ auto function = create_function(factory, "count_at_min", data_types);
+ std::unique_ptr<char[]> memory(new char[function->size_of_data() * 2]);
+ AggregateDataPtr lhs = memory.get();
+ AggregateDataPtr rhs = memory.get() + function->size_of_data();
+ function->create(lhs);
+ function->create(rhs);
+
+ const IColumn* first_columns[1] = {first.get()};
+ const IColumn* second_columns[1] = {second.get()};
+ for (size_t row = 0; row < first->size(); ++row) {
+ function->add(lhs, first_columns, row, arena);
+ }
+ for (size_t row = 0; row < second->size(); ++row) {
+ function->add(rhs, second_columns, row, arena);
+ }
+ function->merge(lhs, rhs, arena);
+
+ ColumnInt64 result;
+ function->insert_result_into(lhs, result);
+ EXPECT_EQ(5, result.get_element(0));
Review Comment:
[P2] Expect two minima in the merge fixture
This fixture selects `count_at_min` over partial inputs `{5,1,5}` and `{5,1,9}`. Each partial has minimum 1 with count 1, so merging equal extrema correctly returns 2. `EXPECT_EQ(5, ...)` therefore fails deterministically with the implementation in this PR. Expect 2, or change the fixture to contain five occurrences of the global minimum.
##########
regression-test/suites/mtmv_p0/ivm/test_ivm_minmax_count.groovy:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.awaitility.Awaitility
+import static java.util.concurrent.TimeUnit.SECONDS
+
+suite("test_ivm_minmax_count") {
+ // The base-table queries below assert the same rows as the MV, so transparent rewrite must be off.
+ sql """set enable_materialized_view_rewrite = false;"""
+ sql """set enable_dml_materialized_view_rewrite = false;"""
+ sql """drop materialized view if exists ivm_mm_count_mv"""
+ sql """drop table if exists ivm_mm_count_t"""
+
+ sql """
+ CREATE TABLE ivm_mm_count_t (
+ id BIGINT NOT NULL,
+ category VARCHAR(16) NOT NULL,
+ score BIGINT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "enable_unique_key_merge_on_write" = "true",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW",
+ "binlog.need_historical_value" = "true"
+ )
+ """
+
+ // category a: min=10 twice, max=30 once
+ // category b: min=5 three times, max=5
+ sql """INSERT INTO ivm_mm_count_t VALUES
+ (1, 'a', 10), (2, 'a', 10), (3, 'a', 30),
+ (4, 'b', 5), (5, 'b', 5), (6, 'b', 5)"""
+
+ sql """
+ CREATE MATERIALIZED VIEW ivm_mm_count_mv
+ BUILD DEFERRED REFRESH INCREMENTAL FALLBACK ON MANUAL
+ KEY(category)
+ DISTRIBUTED BY HASH(category) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ AS SELECT category, MIN(score) AS min_score, MAX(score) AS max_score, COUNT(*) AS row_count
+ FROM ivm_mm_count_t
+ GROUP BY category
+ """
+
+ def assertNoFallback = { task ->
+ def reason = task.IvmFallbackReason
+ assertTrue(reason == null || reason.toString() == "\\N",
+ "unexpected IVM fallback: ${reason}")
+ }
+
+ def latestTask = {
+ def taskResult
+ Awaitility.await().atMost(300, SECONDS).pollInterval(2, SECONDS).until({
+ taskResult = sql_return_maparray("""
+ SELECT Status, RefreshMode, IvmFallbackReason, ErrorMsg
+ FROM tasks('type'='mv')
+ WHERE MvDatabaseName = '${context.dbName}'
+ AND MvName = 'ivm_mm_count_mv'
+ ORDER BY CreateTime DESC, TaskId DESC LIMIT 1
+ """)
+ return !taskResult.isEmpty()
+ && taskResult[0].Status.toString() != 'PENDING'
+ && taskResult[0].Status.toString() != 'RUNNING'
Review Comment:
[P2] Wait for the submitted refresh task before checking fallback
This predicate accepts any latest terminal row without checking its TaskId. During `MTMVTask.onSuccess`, the current task is removed from the running list before `after()` publishes it into MV history, so `tasks()` can temporarily return the preceding completed refresh. At the last-min deletion, this helper can then return the previous INCREMENTAL task and fail the expected COMPLETE assertion; after the smaller insert, it can return the previous fallback task and fail `assertNoFallback`. `Suite.pollMTMVTaskTerminal` already handles this exact gap. Reuse that wait logic and inspect the same completed TaskId, or capture the previous TaskId and wait for a distinct terminal task.
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/agg/IvmAggExtremalProcessor.java:
##########
@@ -109,48 +135,94 @@ public void appendApplyExpressions(IvmAggTarget target, IvmAggApplyContext apply
IvmAggExpressionBuilder ctx = applyContext.expressions();
Slot oldExtreme = applyContext.rawMvSlot(target.getValueStateColumnName());
Expression deltaInsert = applyContext.deltaSlotValue(target, functionKind);
- Expression deltaDel = applyContext.deltaSlotValue(target, deleteSlotRef(target));
- Expression newCount = applyContext.buildNewHiddenCount(target);
-
+ Expression deltaDelete = applyContext.deltaSlotValue(target, deleteSlotRef(target));
+ Expression deltaInsertCount = applyContext.deltaSlotValue(target, insertCountSlotRef(target));
+ Expression deltaDeleteCount = applyContext.deltaSlotValue(target, deleteCountSlotRef(target));
+
+ Slot oldCountSlot = Objects.requireNonNull(target.getHiddenStateSlot(countStateKey),
+ "missing extreme count state for " + target);
+ Expression oldCount = ctx.zeroIfNull(applyContext.rawMvSlot(oldCountSlot.getName()));
+ Expression newTotalCount = applyContext.buildNewHiddenCount(target);
+ Expression zero = new BigIntLiteral(0L);
+
+ // Keep every predicate non-NULL: an empty insert/delete side must read as false, not
+ // as unknown, so the guard below is a plain boolean.
+ Expression insertedPresent = new Not(new IsNull(deltaInsert));
+ Expression deletedPresent = new Not(new IsNull(deltaDelete));
+ Expression insertedBetter = new And(insertedPresent, insertedIsBetter(deltaInsert, oldExtreme));
+ Expression insertedEqual = new And(insertedPresent, new EqualTo(deltaInsert, oldExtreme));
+ Expression deletedEqual = new And(deletedPresent, new EqualTo(deltaDelete, oldExtreme));
+ Expression survivingOldCount = new Subtract(
+ new Add(oldCount, new If(insertedEqual, deltaInsertCount, zero)),
+ new If(deletedEqual, deltaDeleteCount, zero));
+
+ // The old extreme survives when no delete is below it and its remaining count is
+ // positive. An inserted value that beats the old extreme also makes the merge
+ // derivable without a complete refresh.
Expression guardCond = new Or(ImmutableList.of(
- new EqualTo(newCount, new BigIntLiteral(0L)),
- new IsNull(deltaDel),
+ new EqualTo(newTotalCount, zero),
new IsNull(oldExtreme),
- deletedExtremeKeepsOldValueValid(deltaDel, oldExtreme, deltaInsert)
- ));
+ new IsNull(deltaDelete),
+ new And(
+ new And(deletedPresent,
+ deletedExtremeIsAtLeastOld(deltaDelete, oldExtreme)),
+ new Or(ImmutableList.of(
+ insertedBetter,
+ new GreaterThan(survivingOldCount, zero))))));
Expression guard = new AssertTrue(guardCond, new StringLiteral(fallbackMessage()));
+
Expression newExtremeRaw = new CaseWhen(
ImmutableList.of(
- new WhenClause(new EqualTo(newCount, new BigIntLiteral(0L)),
+ new WhenClause(new EqualTo(newTotalCount, zero),
new NullLiteral(oldExtreme.getDataType())),
new WhenClause(new IsNull(oldExtreme), deltaInsert),
- new WhenClause(new IsNull(deltaInsert), oldExtreme)
- ),
- mergeOldAndInsertedExtreme(oldExtreme, deltaInsert)
- );
+ new WhenClause(insertedBetter, deltaInsert)),
+ oldExtreme);
Expression newExtremeGuarded = new If(guard, newExtremeRaw,
new NullLiteral(newExtremeRaw.getDataType()));
- applyContext.putFinalExpression(target, target.getHiddenStateSlot(IvmAggStateKey.COUNT).getName(), newCount);
+ Expression newCountRaw = new CaseWhen(
+ ImmutableList.of(
+ new WhenClause(new EqualTo(newTotalCount, zero), zero),
+ new WhenClause(new IsNull(oldExtreme), deltaInsertCount),
Review Comment:
[P1] Subtract matching deletes when initializing the extreme count
When `oldExtreme` is NULL, persisting `deltaInsertCount` ignores canceled delete contributions. A reachable plan is `Aggregate(MIN(L.v)) -> InnerJoin(L.k = R.k)`: initially L has `(k=0,v=NULL)` and R has keys 0,1,2,3. Before refresh, insert L rows `(1,1),(2,1),(3,2)` and delete R key 1. The join rewrite emits `+{1,1,2}` from `leftDelta JOIN rightPre` and `-{1}` from `leftPost JOIN rightDelta`. The resulting MIN is correctly 1, but this branch stores MIN_COUNT=2 instead of 1. On the next refresh, deleting L key 2 leaves only non-NULL value 2; `survivingOldCount=2-1` stays positive, so the guard silently retains MIN=1 instead of falling back. This later wrong result is introduced by the new count state; the previous guard would reject that boundary deletion. Initialize the count from net contributions at the selected extreme and preserve fallback when its survival cannot be proved. MAX has the symmetric failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281849996
<details>
<summary>TPC-H: <b>Total hot run time: 28259 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fe394a7d126524423d42188559245ad083f7f3c6, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17603 5170 5030 5030
q2 2103 324 253 253
q3 10236 1365 797 797
q4 4683 465 332 332
q5 7421 792 526 526
q6 179 186 152 152
q7 760 795 595 595
q8 9248 1312 1128 1128
q9 5403 4174 4176 4174
q10 6823 1279 1039 1039
q11 424 246 226 226
q12 639 392 274 274
q13 18105 3572 2962 2962
q14 275 269 245 245
q15 q16 724 712 645 645
q17 1199 742 582 582
q18 6982 6170 6108 6108
q19 1092 986 606 606
q20 366 326 225 225
q21 4974 2098 2301 2098
q22 350 293 262 262
Total cold run time: 99589 ms
Total hot run time: 28259 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5588 5573 5595 5573
q2 689 587 510 510
q3 5229 5395 4906 4906
q4 1976 2178 1527 1527
q5 5228 5137 5417 5137
q6 245 192 143 143
q7 2331 2057 1839 1839
q8 3002 2657 2660 2657
q9 7973 7889 8026 7889
q10 3657 3710 3226 3226
q11 786 501 428 428
q12 721 658 538 538
q13 3482 3842 3266 3266
q14 331 332 282 282
q15 q16 714 723 643 643
q17 8164 7315 6868 6868
q18 13008 12185 12849 12185
q19 907 788 784 784
q20 2133 2130 1922 1922
q21 5510 4631 4674 4631
q22 517 458 417 417
Total cold run time: 72191 ms
Total hot run time: 65371 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281385984
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281973497
<details>
<summary>TPC-DS: <b>Total hot run time: 150979 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fe394a7d126524423d42188559245ad083f7f3c6, data reload: false
query5 4301 583 470 470
query6 422 226 197 197
query7 4798 473 225 225
query8 320 166 152 152
query9 8762 3131 3165 3131
query10 486 319 264 264
query11 6031 3830 3574 3574
query12 149 90 88 88
query13 1248 460 325 325
query14 6145 3956 3670 3670
query14_1 3478 3436 3452 3436
query15 211 197 177 177
query16 941 436 410 410
query17 866 665 542 542
query18 2419 435 319 319
query19 203 163 134 134
query20 85 84 83 83
query21 208 141 111 111
query22 12971 12974 12972 12972
query23 13348 12779 12320 12320
query23_1 12531 12534 12550 12534
query24 7362 826 461 461
query24_1 483 474 481 474
query25 533 424 356 356
query26 1258 252 138 138
query27 2776 446 263 263
query28 4635 1945 1929 1929
query29 1587 609 459 459
query30 299 217 186 186
query31 1018 931 803 803
query32 148 97 95 95
query33 539 302 253 253
query34 919 846 508 508
query35 731 776 646 646
query36 828 793 732 732
query37 132 101 81 81
query38 1803 1742 1668 1668
query39 809 803 740 740
query39_1 727 736 736 736
query40 213 118 100 100
query41 72 71 70 70
query42 91 93 83 83
query43 346 375 313 313
query44 1340 680 688 680
query45 178 179 165 165
query46 829 963 562 562
query47 3020 2969 2913 2913
query48 303 313 226 226
query49 589 403 329 329
query50 704 273 206 206
query51 10492 10246 10359 10246
query52 76 81 71 71
query53 186 203 149 149
query54 235 190 195 190
query55 75 72 67 67
query56 254 209 221 209
query57 1670 1646 1557 1557
query58 290 264 256 256
query59 2070 2127 1914 1914
query60 281 241 226 226
query61 150 138 153 138
query62 381 327 294 294
query63 195 156 161 156
query64 2755 982 827 827
query65 3332 3281 3264 3264
query66 1785 419 294 294
query67 19959 20084 19889 19889
query68 2924 1010 611 611
query69 385 298 243 243
query70 846 771 769 769
query71 303 241 215 215
query72 2752 2375 2282 2282
query73 509 533 294 294
query74 4661 4529 4347 4347
query75 2343 2271 1922 1922
query76 2316 1040 614 614
query77 346 374 268 268
query78 9771 9735 9214 9214
query79 1100 847 519 519
query80 1229 421 352 352
query81 594 321 270 270
query82 581 137 108 108
query83 305 198 180 180
query84 313 122 102 102
query85 883 445 373 373
query86 484 238 230 230
query87 1986 1943 1792 1792
query88 3761 2682 2642 2642
query89 331 282 242 242
query90 1863 167 158 158
query91 158 156 127 127
query92 96 90 89 89
query93 1138 989 572 572
query94 660 306 270 270
query95 563 325 359 325
query96 663 490 233 233
query97 2426 2432 2372 2372
query98 163 152 147 147
query99 762 766 634 634
Total cold run time: 235242 ms
Total hot run time: 150979 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6287117595
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6266818367
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69286:
URL: https://github.com/apache/doris/pull/69286#discussion_r4314470270
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/agg/IvmAggExtremalProcessor.java:
##########
@@ -109,48 +135,94 @@ public void appendApplyExpressions(IvmAggTarget target, IvmAggApplyContext apply
IvmAggExpressionBuilder ctx = applyContext.expressions();
Slot oldExtreme = applyContext.rawMvSlot(target.getValueStateColumnName());
Expression deltaInsert = applyContext.deltaSlotValue(target, functionKind);
- Expression deltaDel = applyContext.deltaSlotValue(target, deleteSlotRef(target));
- Expression newCount = applyContext.buildNewHiddenCount(target);
-
+ Expression deltaDelete = applyContext.deltaSlotValue(target, deleteSlotRef(target));
+ Expression deltaInsertCount = applyContext.deltaSlotValue(target, insertCountSlotRef(target));
+ Expression deltaDeleteCount = applyContext.deltaSlotValue(target, deleteCountSlotRef(target));
+
+ Slot oldCountSlot = Objects.requireNonNull(target.getHiddenStateSlot(countStateKey),
+ "missing extreme count state for " + target);
+ Expression oldCount = ctx.zeroIfNull(applyContext.rawMvSlot(oldCountSlot.getName()));
+ Expression newTotalCount = applyContext.buildNewHiddenCount(target);
+ Expression zero = new BigIntLiteral(0L);
+
+ // Keep every predicate non-NULL: an empty insert/delete side must read as false, not
+ // as unknown, so the guard below is a plain boolean.
+ Expression insertedPresent = new Not(new IsNull(deltaInsert));
+ Expression deletedPresent = new Not(new IsNull(deltaDelete));
+ Expression insertedBetter = new And(insertedPresent, insertedIsBetter(deltaInsert, oldExtreme));
+ Expression insertedEqual = new And(insertedPresent, new EqualTo(deltaInsert, oldExtreme));
+ Expression deletedEqual = new And(deletedPresent, new EqualTo(deltaDelete, oldExtreme));
+ Expression survivingOldCount = new Subtract(
+ new Add(oldCount, new If(insertedEqual, deltaInsertCount, zero)),
+ new If(deletedEqual, deltaDeleteCount, zero));
+
+ // The old extreme survives when no delete is below it and its remaining count is
+ // positive. An inserted value that beats the old extreme also makes the merge
+ // derivable without a complete refresh.
Expression guardCond = new Or(ImmutableList.of(
- new EqualTo(newCount, new BigIntLiteral(0L)),
- new IsNull(deltaDel),
+ new EqualTo(newTotalCount, zero),
new IsNull(oldExtreme),
- deletedExtremeKeepsOldValueValid(deltaDel, oldExtreme, deltaInsert)
- ));
+ new IsNull(deltaDelete),
+ new And(
+ new And(deletedPresent,
+ deletedExtremeIsAtLeastOld(deltaDelete, oldExtreme)),
+ new Or(ImmutableList.of(
+ insertedBetter,
+ new GreaterThan(survivingOldCount, zero))))));
Expression guard = new AssertTrue(guardCond, new StringLiteral(fallbackMessage()));
+
Expression newExtremeRaw = new CaseWhen(
ImmutableList.of(
- new WhenClause(new EqualTo(newCount, new BigIntLiteral(0L)),
+ new WhenClause(new EqualTo(newTotalCount, zero),
new NullLiteral(oldExtreme.getDataType())),
new WhenClause(new IsNull(oldExtreme), deltaInsert),
- new WhenClause(new IsNull(deltaInsert), oldExtreme)
- ),
- mergeOldAndInsertedExtreme(oldExtreme, deltaInsert)
- );
+ new WhenClause(insertedBetter, deltaInsert)),
+ oldExtreme);
Expression newExtremeGuarded = new If(guard, newExtremeRaw,
new NullLiteral(newExtremeRaw.getDataType()));
- applyContext.putFinalExpression(target, target.getHiddenStateSlot(IvmAggStateKey.COUNT).getName(), newCount);
+ Expression newCountRaw = new CaseWhen(
+ ImmutableList.of(
+ new WhenClause(new EqualTo(newTotalCount, zero), zero),
+ new WhenClause(new IsNull(oldExtreme), deltaInsertCount),
Review Comment:
Fixed in 87a8d552ab5. When the old extreme is NULL, the count now comes from the net insert/delete contributions at the insert-side extreme: subtract deltaDeleteCount when the delete-side extreme equals it, use deltaInsertCount when the delete extreme is farther from the aggregate extreme, and keep the complete-refresh fallback when a delete is closer or cancels every inserted row. I also added the two-refresh canceled-join regression test_ivm_minmax_count_join, which reproduced the stale MIN=1 behavior before the fix.
##########
be/test/exprs/aggregate/agg_count_at_min_max_test.cpp:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include <cstring>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "core/column/column.h"
+#include "core/column/column_string.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+#include "exprs/aggregate/aggregate_function_simple_factory.h"
+#include "exprs/aggregate/single_value_data.h"
+
+namespace doris {
+
+void register_aggregate_function_count_at_min_max(AggregateFunctionSimpleFactory& factory);
+
+namespace {
+
+AggregateFunctionPtr create_function(AggregateFunctionSimpleFactory& factory,
+ const std::string& name, const DataTypes& argument_types) {
+ auto function = factory.get(name, argument_types, nullptr, false, -1);
+ EXPECT_NE(function, nullptr);
+ return function;
+}
+
+void aggregate_int32(IAggregateFunction* function, Arena& arena, const IColumn& column,
+ int64_t expected) {
+ std::unique_ptr<char[]> memory(new char[function->size_of_data()]);
+ AggregateDataPtr place = memory.get();
+ function->create(place);
+ const IColumn* columns[1] = {&column};
+ for (size_t row = 0; row < column.size(); ++row) {
+ function->add(place, columns, row, arena);
+ }
+ ColumnInt64 result;
+ function->insert_result_into(place, result);
+ EXPECT_EQ(expected, result.get_element(0));
+ function->destroy(place);
+}
+
+} // namespace
+
+TEST(CountAtMinMaxTest, Int32) {
+ AggregateFunctionSimpleFactory factory;
+ register_aggregate_function_count_at_min_max(factory);
+ DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+
+ auto column = ColumnInt32::create();
+ for (int value : {1, 2, 1, 3, 1, 2}) {
+ column->insert_value(value);
+ }
+
+ Arena arena;
+ auto count_at_min = create_function(factory, "count_at_min", data_types);
+ aggregate_int32(count_at_min.get(), arena, *column, 3);
+
+ auto count_at_max = create_function(factory, "count_at_max", data_types);
+ aggregate_int32(count_at_max.get(), arena, *column, 1);
+}
+
+TEST(CountAtMinMaxTest, EmptyInput) {
+ AggregateFunctionSimpleFactory factory;
+ register_aggregate_function_count_at_min_max(factory);
+ DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+ Arena arena;
+
+ auto column = ColumnInt32::create();
+ for (const auto& name : {"count_at_min", "count_at_max"}) {
+ auto function = create_function(factory, name, data_types);
+ aggregate_int32(function.get(), arena, *column, 0);
+ }
+}
+
+TEST(CountAtMinMaxTest, Merge) {
+ AggregateFunctionSimpleFactory factory;
+ register_aggregate_function_count_at_min_max(factory);
+ DataTypes data_types = {std::make_shared<DataTypeInt32>()};
+ Arena arena;
+
+ auto first = ColumnInt32::create();
+ auto second = ColumnInt32::create();
+ for (int value : {5, 1, 5}) {
+ first->insert_value(value);
+ }
+ for (int value : {5, 1, 9}) {
+ second->insert_value(value);
+ }
+
+ auto function = create_function(factory, "count_at_min", data_types);
+ std::unique_ptr<char[]> memory(new char[function->size_of_data() * 2]);
+ AggregateDataPtr lhs = memory.get();
+ AggregateDataPtr rhs = memory.get() + function->size_of_data();
+ function->create(lhs);
+ function->create(rhs);
+
+ const IColumn* first_columns[1] = {first.get()};
+ const IColumn* second_columns[1] = {second.get()};
+ for (size_t row = 0; row < first->size(); ++row) {
+ function->add(lhs, first_columns, row, arena);
+ }
+ for (size_t row = 0; row < second->size(); ++row) {
+ function->add(rhs, second_columns, row, arena);
+ }
+ function->merge(lhs, rhs, arena);
+
+ ColumnInt64 result;
+ function->insert_result_into(lhs, result);
+ EXPECT_EQ(5, result.get_element(0));
Review Comment:
Fixed in 87a8d552ab5. The merge fixture now expects 2: each partial has minimum 1 with count 1, so merging equal extrema returns 2.
##########
regression-test/suites/mtmv_p0/ivm/test_ivm_minmax_count.groovy:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.awaitility.Awaitility
+import static java.util.concurrent.TimeUnit.SECONDS
+
+suite("test_ivm_minmax_count") {
+ // The base-table queries below assert the same rows as the MV, so transparent rewrite must be off.
+ sql """set enable_materialized_view_rewrite = false;"""
+ sql """set enable_dml_materialized_view_rewrite = false;"""
+ sql """drop materialized view if exists ivm_mm_count_mv"""
+ sql """drop table if exists ivm_mm_count_t"""
+
+ sql """
+ CREATE TABLE ivm_mm_count_t (
+ id BIGINT NOT NULL,
+ category VARCHAR(16) NOT NULL,
+ score BIGINT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "enable_unique_key_merge_on_write" = "true",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW",
+ "binlog.need_historical_value" = "true"
+ )
+ """
+
+ // category a: min=10 twice, max=30 once
+ // category b: min=5 three times, max=5
+ sql """INSERT INTO ivm_mm_count_t VALUES
+ (1, 'a', 10), (2, 'a', 10), (3, 'a', 30),
+ (4, 'b', 5), (5, 'b', 5), (6, 'b', 5)"""
+
+ sql """
+ CREATE MATERIALIZED VIEW ivm_mm_count_mv
+ BUILD DEFERRED REFRESH INCREMENTAL FALLBACK ON MANUAL
+ KEY(category)
+ DISTRIBUTED BY HASH(category) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ AS SELECT category, MIN(score) AS min_score, MAX(score) AS max_score, COUNT(*) AS row_count
+ FROM ivm_mm_count_t
+ GROUP BY category
+ """
+
+ def assertNoFallback = { task ->
+ def reason = task.IvmFallbackReason
+ assertTrue(reason == null || reason.toString() == "\\N",
+ "unexpected IVM fallback: ${reason}")
+ }
+
+ def latestTask = {
+ def taskResult
+ Awaitility.await().atMost(300, SECONDS).pollInterval(2, SECONDS).until({
+ taskResult = sql_return_maparray("""
+ SELECT Status, RefreshMode, IvmFallbackReason, ErrorMsg
+ FROM tasks('type'='mv')
+ WHERE MvDatabaseName = '${context.dbName}'
+ AND MvName = 'ivm_mm_count_mv'
+ ORDER BY CreateTime DESC, TaskId DESC LIMIT 1
+ """)
+ return !taskResult.isEmpty()
+ && taskResult[0].Status.toString() != 'PENDING'
+ && taskResult[0].Status.toString() != 'RUNNING'
Review Comment:
Fixed in 87a8d552ab5. Both MIN/MAX count suites now remember the previous TaskId and only accept a terminal row whose TaskId differs, so the running-list-to-history gap can no longer return the preceding refresh. The new join suite uses the same wait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6268543523
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6271983887
# FE Regression Coverage Report
Increment line coverage ` 79.37% (100/126)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6272882368
# FE Regression Coverage Report
Increment line coverage ` 79.37% (100/126)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69286_87a8d552ab57c552322af4ce4a77ca290d6829ac_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282048615
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282057620
Codex automated review failed and did not complete.
Error: Review context preparation failed before Codex ran; inspect the 'Prepare authoritative PR context and required AGENTS guides' step.
Workflow run: https://github.com/apache/doris/actions/runs/39358482180
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6281321162
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Track MIN/MAX extreme counts for incremental refresh [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69286:
URL: https://github.com/apache/doris/pull/69286#issuecomment-6282447718
<details>
<summary>TPC-DS: <b>Total hot run time: 150970 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 008d08b586b9e981e19d07e8e5ca01d63f244045, data reload: false
query5 4294 586 446 446
query6 428 206 191 191
query7 4819 440 236 236
query8 326 161 146 146
query9 8760 3159 3171 3159
query10 485 320 282 282
query11 6047 3838 3594 3594
query12 146 92 89 89
query13 1254 450 338 338
query14 6078 3930 3620 3620
query14_1 3446 3374 3424 3374
query15 211 201 179 179
query16 972 468 468 468
query17 876 666 545 545
query18 2424 443 320 320
query19 196 167 136 136
query20 88 85 89 85
query21 215 134 108 108
query22 12999 12999 12822 12822
query23 13422 12887 12531 12531
query23_1 12616 12474 12507 12474
query24 7388 840 463 463
query24_1 481 495 476 476
query25 550 420 365 365
query26 1252 269 143 143
query27 2760 439 272 272
query28 4638 1943 1948 1943
query29 1575 610 490 490
query30 320 227 188 188
query31 1011 935 822 822
query32 162 92 94 92
query33 537 316 245 245
query34 944 840 506 506
query35 752 783 647 647
query36 827 785 730 730
query37 134 101 87 87
query38 1797 1741 1686 1686
query39 795 830 741 741
query39_1 729 725 732 725
query40 223 116 99 99
query41 75 68 71 68
query42 88 89 92 89
query43 354 362 317 317
query44 1379 687 697 687
query45 183 177 160 160
query46 845 967 566 566
query47 2961 3058 2816 2816
query48 298 318 213 213
query49 570 399 297 297
query50 700 262 206 206
query51 10191 10439 10060 10060
query52 75 81 68 68
query53 191 204 150 150
query54 253 225 183 183
query55 72 71 63 63
query56 229 208 207 207
query57 1638 1618 1565 1565
query58 292 265 252 252
query59 2104 2145 1921 1921
query60 294 237 224 224
query61 145 147 145 145
query62 385 333 287 287
query63 199 163 162 162
query64 2744 944 744 744
query65 3299 3285 3284 3284
query66 1785 421 306 306
query67 20355 20224 20050 20050
query68 2973 1007 599 599
query69 387 281 236 236
query70 894 775 760 760
query71 304 232 214 214
query72 2709 2558 2543 2543
query73 518 543 301 301
query74 4646 4533 4373 4373
query75 2341 2295 1909 1909
query76 2320 1029 622 622
query77 352 366 275 275
query78 9940 9806 9104 9104
query79 953 899 516 516
query80 830 420 335 335
query81 580 319 271 271
query82 574 131 111 111
query83 305 194 185 185
query84 307 119 104 104
query85 897 436 368 368
query86 360 244 230 230
query87 1987 1971 1801 1801
query88 3686 2680 2650 2650
query89 343 284 237 237
query90 1808 171 171 171
query91 174 155 128 128
query92 94 87 86 86
query93 1005 991 570 570
query94 514 306 271 271
query95 581 386 308 308
query96 680 495 230 230
query97 2457 2436 2337 2337
query98 165 152 146 146
query99 765 760 660 660
Total cold run time: 234809 ms
Total hot run time: 150970 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org