You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2025/12/10 04:08:14 UTC
[PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
morningman opened a new pull request, #58889:
URL: https://github.com/apache/doris/pull/58889
### What problem does this PR solve?
Related PR: #44039
Problem Summary:
In #44039, we disable the count push down to iceberg table if it has position delete file,
because there may be dangling delete file that will cause the statistics info incorrect.
This PR add a new session variable `ignore_iceberg_dangling_delete`, default is false,
when set to true, Doris will ignore the dangling delete issue and do count push down for iceberg table,
add user should take care of the result.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58889:
URL: https://github.com/apache/doris/pull/58889#issuecomment-3637468848
# FE Regression Coverage Report
Increment line coverage ` 100.00% (8/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58889_a6bb8511e5260b043c23444ca19867b87f91951e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58889_a6bb8511e5260b043c23444ca19867b87f91951e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58889:
URL: https://github.com/apache/doris/pull/58889#issuecomment-3652655408
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #58889:
URL: https://github.com/apache/doris/pull/58889#issuecomment-3635389224
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58889:
URL: https://github.com/apache/doris/pull/58889#issuecomment-3636022799
# FE Regression Coverage Report
Increment line coverage ` 100.00% (8/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58889_a6bb8511e5260b043c23444ca19867b87f91951e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58889_a6bb8511e5260b043c23444ca19867b87f91951e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #58889:
URL: https://github.com/apache/doris/pull/58889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58889:
URL: https://github.com/apache/doris/pull/58889#issuecomment-3635710762
# FE UT Coverage Report
Increment line coverage ` 12.50% (1/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58889_a6bb8511e5260b043c23444ca19867b87f91951e/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58889_a6bb8511e5260b043c23444ca19867b87f91951e/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #58889:
URL: https://github.com/apache/doris/pull/58889#issuecomment-3635290471
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58889:
URL: https://github.com/apache/doris/pull/58889#issuecomment-3635471619
<details>
<summary>TPC-H: <b>Total hot run time: 35634 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a6bb8511e5260b043c23444ca19867b87f91951e, data reload: false
------ Round 1 ----------------------------------
q1 17615 4256 4070 4070
q2 2027 366 240 240
q3 10156 1329 760 760
q4 10265 798 312 312
q5 7561 2158 1945 1945
q6 186 172 142 142
q7 1016 850 715 715
q8 9353 1455 1120 1120
q9 7031 5326 5352 5326
q10 6777 2392 1997 1997
q11 527 321 294 294
q12 684 742 576 576
q13 17768 3739 3072 3072
q14 294 302 270 270
q15 586 523 508 508
q16 941 931 866 866
q17 735 774 561 561
q18 7914 7258 7139 7139
q19 1123 971 620 620
q20 413 367 252 252
q21 4313 3951 3862 3862
q22 1037 992 987 987
Total cold run time: 108322 ms
Total hot run time: 35634 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4112 4046 4060 4046
q2 329 426 307 307
q3 2148 2715 2323 2323
q4 1315 1722 1299 1299
q5 4249 4849 4739 4739
q6 229 180 130 130
q7 2054 1972 1822 1822
q8 2648 2584 2514 2514
q9 7943 7499 7671 7499
q10 3026 3245 2819 2819
q11 607 507 493 493
q12 703 734 623 623
q13 3584 3898 3331 3331
q14 288 293 295 293
q15 581 514 509 509
q16 892 915 936 915
q17 1323 1415 1498 1415
q18 7894 7583 7632 7583
q19 870 855 925 855
q20 2051 2122 1955 1955
q21 4678 4191 4139 4139
q22 1091 1047 1011 1011
Total cold run time: 52615 ms
Total hot run time: 50620 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58889:
URL: https://github.com/apache/doris/pull/58889#issuecomment-3635499013
<details>
<summary>TPC-DS: <b>Total hot run time: 181181 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a6bb8511e5260b043c23444ca19867b87f91951e, data reload: false
query5 5340 651 482 482
query6 324 226 210 210
query7 4232 465 274 274
query8 303 252 239 239
query9 8778 2572 2591 2572
query10 540 373 340 340
query11 15466 15259 14605 14605
query12 179 120 116 116
query13 1277 488 399 399
query14 6239 3234 2912 2912
query14_1 2877 2856 2829 2829
query15 214 208 184 184
query16 805 450 473 450
query17 1094 730 572 572
query18 2542 435 340 340
query19 236 232 215 215
query20 123 114 114 114
query21 224 141 115 115
query22 4179 4239 4065 4065
query23 16569 15950 15964 15950
query23_1 15977 16046 16020 16020
query24 7423 1681 1232 1232
query24_1 1221 1251 1215 1215
query25 558 468 414 414
query26 1263 269 157 157
query27 2782 453 316 316
query28 4486 2159 2150 2150
query29 818 545 434 434
query30 309 239 217 217
query31 812 686 638 638
query32 77 68 67 67
query33 539 322 291 291
query34 915 913 543 543
query35 802 804 732 732
query36 863 908 831 831
query37 128 90 79 79
query38 3926 3809 3754 3754
query39 749 750 708 708
query39_1 704 697 705 697
query40 229 144 126 126
query41 73 70 68 68
query42 108 106 108 106
query43 430 429 404 404
query44 1321 762 778 762
query45 194 192 187 187
query46 879 994 633 633
query47 1732 1716 1634 1634
query48 336 336 268 268
query49 659 456 370 370
query50 656 310 226 226
query51 3807 3871 3786 3786
query52 110 115 103 103
query53 321 362 300 300
query54 312 286 282 282
query55 77 77 71 71
query56 311 313 302 302
query57 1163 1152 1077 1077
query58 292 272 259 259
query59 2417 2582 2390 2390
query60 345 327 317 317
query61 188 190 189 189
query62 695 663 616 616
query63 331 298 300 298
query64 5133 1442 1128 1128
query65 4018 3925 3957 3925
query66 1444 468 334 334
query67 15369 15179 14944 14944
query68 7888 1028 752 752
query69 499 354 318 318
query70 1078 1052 1017 1017
query71 374 310 293 293
query72 6067 4972 5143 4972
query73 654 584 304 304
query74 8845 8708 8599 8599
query75 3549 3501 3132 3132
query76 3887 1144 753 753
query77 520 413 290 290
query78 9539 9861 8832 8832
query79 1417 881 600 600
query80 735 657 569 569
query81 524 276 236 236
query82 208 128 103 103
query83 270 262 241 241
query84 257 120 97 97
query85 905 517 472 472
query86 376 295 278 278
query87 4093 3984 3918 3918
query88 3177 2289 2300 2289
query89 477 441 396 396
query90 2241 162 148 148
query91 174 168 146 146
query92 86 70 63 63
query93 1224 925 574 574
query94 488 308 278 278
query95 588 339 370 339
query96 605 468 206 206
query97 2600 2647 2605 2605
query98 218 195 198 195
query99 1271 1288 1245 1245
Total cold run time: 263486 ms
Total hot run time: 181181 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58889:
URL: https://github.com/apache/doris/pull/58889#issuecomment-3635509393
<details>
<summary>ClickBench: <b>Total hot run time: 27.47 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a6bb8511e5260b043c23444ca19867b87f91951e, data reload: false
query1 0.05 0.04 0.04
query2 0.10 0.05 0.05
query3 0.26 0.09 0.08
query4 1.62 0.11 0.11
query5 0.28 0.26 0.26
query6 1.16 0.65 0.64
query7 0.03 0.03 0.03
query8 0.05 0.04 0.05
query9 0.57 0.51 0.52
query10 0.55 0.55 0.55
query11 0.16 0.10 0.12
query12 0.15 0.11 0.13
query13 0.62 0.61 0.61
query14 1.00 0.99 0.98
query15 0.81 0.81 0.82
query16 0.42 0.40 0.41
query17 1.09 1.07 1.02
query18 0.23 0.21 0.21
query19 1.86 1.90 1.82
query20 0.02 0.01 0.02
query21 15.43 0.29 0.14
query22 4.67 0.05 0.05
query23 16.03 0.28 0.10
query24 2.26 0.59 0.27
query25 0.08 0.08 0.08
query26 0.14 0.13 0.14
query27 0.07 0.06 0.05
query28 3.78 1.23 1.02
query29 12.65 4.06 3.34
query30 0.28 0.15 0.16
query31 2.82 0.63 0.39
query32 3.24 0.56 0.46
query33 3.02 2.98 2.98
query34 16.86 5.18 4.50
query35 4.61 4.61 4.58
query36 0.65 0.51 0.50
query37 0.10 0.07 0.06
query38 0.07 0.04 0.04
query39 0.05 0.03 0.03
query40 0.17 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 98.18 s
Total hot run time: 27.47 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](iceberg) able to ignore iceberg dangling delete to do count push down [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58889:
URL: https://github.com/apache/doris/pull/58889#issuecomment-3652655479
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org