You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xinyiZzz (via GitHub)" <gi...@apache.org> on 2024/10/24 11:02:59 UTC
[PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
xinyiZzz opened a new pull request, #42421:
URL: https://github.com/apache/doris/pull/42421
1. A query_id is randomly generated to replace t_query_plan_info.query_id. external query does not need to report anything to FE, so the query_id can be changed.
Otherwise, multiple independent concurrent open tablet scanners have the same query_id. when one of the scanners ends, the other scanners will be canceled through FragmentMgr.cancel(query_id).
2. (query_id, fragment_id) is executed only on one BE, locks _pipeline_map.
3. External query (flink/spark read tablets) not need to report to FE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2440370887
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2436762771
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2436759342
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2436871391
TeamCity be ut coverage result:
Function Coverage: 37.46% (9713/25932)
Line Coverage: 28.71% (80536/280548)
Region Coverage: 28.13% (41642/148021)
Branch Coverage: 24.69% (21157/85690)
Coverage Report: http://coverage.selectdb-in.cc/coverage/26e37053ce3f8ff8b08c7842293a61f16cc0e70d_26e37053ce3f8ff8b08c7842293a61f16cc0e70d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2434975953
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "wanggx (via GitHub)" <gi...@apache.org>.
wanggx commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-3400998480
> 1. A query_id is randomly generated to replace t_query_plan_info.query_id. external query does not need to report anything to FE, so the query_id can be changed.
> Otherwise, multiple independent concurrent open tablet scanners have the same query_id. when one of the scanners ends, the other scanners will be canceled through FragmentMgr.cancel(query_id).
> 2. (query_id, fragment_id) is executed only on one BE, locks _pipeline_map.
> 3. External query (flink/spark read tablets) not need to report to FE.
according to your describe，when multiple independent concurrent open tablet scanners ，must cause an error，why has this problem，I cannot understand it .
https://doris-forum.org.cn/questions/D15j4/tong-guo-de-fang-shi-fang-wen-biao-ti-shi-bao-cuo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2437186259
<details>
<summary>ClickBench: <b>Total hot run time: 32.86 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3075d09fbf919584fefcc795b3b3b8816c8ce87d, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.03 0.03
query3 0.23 0.06 0.07
query4 1.65 0.11 0.10
query5 0.42 0.41 0.41
query6 1.17 0.67 0.64
query7 0.03 0.02 0.02
query8 0.04 0.03 0.04
query9 0.57 0.50 0.50
query10 0.56 0.54 0.54
query11 0.14 0.11 0.11
query12 0.14 0.11 0.11
query13 0.60 0.59 0.61
query14 2.82 2.72 2.73
query15 0.91 0.83 0.85
query16 0.40 0.38 0.38
query17 1.09 1.06 1.05
query18 0.20 0.20 0.20
query19 1.93 1.88 2.03
query20 0.01 0.01 0.01
query21 15.36 0.59 0.57
query22 2.26 1.99 1.81
query23 17.19 1.05 0.79
query24 2.51 1.80 1.18
query25 0.33 0.21 0.12
query26 0.37 0.13 0.13
query27 0.04 0.04 0.04
query28 10.26 1.10 1.08
query29 12.57 3.35 3.30
query30 0.25 0.08 0.06
query31 2.87 0.39 0.39
query32 3.25 0.46 0.46
query33 2.98 2.99 3.02
query34 16.96 4.46 4.48
query35 4.62 4.56 4.48
query36 0.68 0.47 0.47
query37 0.09 0.06 0.06
query38 0.04 0.03 0.03
query39 0.03 0.02 0.02
query40 0.17 0.13 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 105.98 s
Total hot run time: 32.86 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2440370941
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2434965091
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #42421:
URL: https://github.com/apache/doris/pull/42421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2436764210
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2436760967
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2434965031
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2437090320
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2437155867
TeamCity be ut coverage result:
Function Coverage: 37.45% (9711/25932)
Line Coverage: 28.71% (80549/280551)
Region Coverage: 28.13% (41637/148021)
Branch Coverage: 24.68% (21150/85690)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3075d09fbf919584fefcc795b3b3b8816c8ce87d_3075d09fbf919584fefcc795b3b3b8816c8ce87d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2435027036
TeamCity be ut coverage result:
Function Coverage: 37.47% (9713/25923)
Line Coverage: 28.74% (80577/280343)
Region Coverage: 28.16% (41651/147893)
Branch Coverage: 24.73% (21160/85566)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7846e1e11809abe0fda8e83193270c2e3c6e47f8_7846e1e11809abe0fda8e83193270c2e3c6e47f8/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2435048590
<details>
<summary>TPC-H: <b>Total hot run time: 41223 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7846e1e11809abe0fda8e83193270c2e3c6e47f8, data reload: false
------ Round 1 ----------------------------------
q1 17583 7487 7424 7424
q2 2011 284 269 269
q3 12244 1045 1183 1045
q4 10561 855 796 796
q5 7776 3114 3094 3094
q6 243 155 152 152
q7 1072 618 605 605
q8 9380 1975 2007 1975
q9 6623 6484 6411 6411
q10 7056 2425 2419 2419
q11 452 246 262 246
q12 418 221 214 214
q13 17801 3028 2996 2996
q14 245 213 215 213
q15 566 515 516 515
q16 627 591 574 574
q17 970 466 502 466
q18 7274 6577 6664 6577
q19 1346 1108 1022 1022
q20 505 179 184 179
q21 4018 3219 3044 3044
q22 1134 987 993 987
Total cold run time: 109905 ms
Total hot run time: 41223 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7346 7306 7290 7290
q2 325 231 229 229
q3 3075 2940 2960 2940
q4 2182 1856 1789 1789
q5 5788 5785 5765 5765
q6 235 147 145 145
q7 2302 1839 1779 1779
q8 3420 3524 3450 3450
q9 8891 8925 8918 8918
q10 3594 3579 3596 3579
q11 599 500 473 473
q12 843 643 594 594
q13 12185 3211 3216 3211
q14 314 276 276 276
q15 581 541 536 536
q16 700 645 648 645
q17 1853 1620 1625 1620
q18 8236 7946 7523 7523
q19 1725 1340 1513 1340
q20 2126 1903 1895 1895
q21 5690 5423 5423 5423
q22 1151 1067 1047 1047
Total cold run time: 73161 ms
Total hot run time: 60467 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2435073261
<details>
<summary>TPC-DS: <b>Total hot run time: 193111 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7846e1e11809abe0fda8e83193270c2e3c6e47f8, data reload: false
query1 920 419 428 419
query2 6269 2150 2059 2059
query3 8685 195 208 195
query4 35012 23965 23678 23678
query5 5414 474 469 469
query6 276 169 164 164
query7 4205 301 295 295
query8 282 233 237 233
query9 9319 2742 2709 2709
query10 485 274 284 274
query11 18330 15472 15265 15265
query12 156 111 99 99
query13 1572 430 413 413
query14 10064 7489 7497 7489
query15 251 170 179 170
query16 8050 460 514 460
query17 1621 592 588 588
query18 2181 312 327 312
query19 374 159 169 159
query20 131 117 120 117
query21 216 116 117 116
query22 4767 4621 4516 4516
query23 34849 34075 34291 34075
query24 9089 2771 2783 2771
query25 626 410 407 407
query26 720 156 166 156
query27 2028 277 302 277
query28 6385 2435 2431 2431
query29 763 431 430 430
query30 261 166 166 166
query31 1056 810 798 798
query32 101 56 58 56
query33 659 307 318 307
query34 945 516 529 516
query35 1042 903 885 885
query36 1096 932 968 932
query37 148 89 93 89
query38 4398 4322 4327 4322
query39 1653 1462 1403 1403
query40 216 100 100 100
query41 49 47 47 47
query42 127 106 102 102
query43 544 502 493 493
query44 1185 808 807 807
query45 200 167 164 164
query46 1134 697 711 697
query47 1932 1826 1818 1818
query48 434 318 315 315
query49 856 440 442 440
query50 805 406 404 404
query51 7146 6943 6887 6887
query52 103 93 91 91
query53 257 182 182 182
query54 1014 458 418 418
query55 81 80 75 75
query56 281 260 256 256
query57 1298 1158 1148 1148
query58 227 273 239 239
query59 3239 2985 3031 2985
query60 298 264 267 264
query61 110 98 102 98
query62 838 686 680 680
query63 213 187 189 187
query64 3607 633 605 605
query65 3308 3220 3213 3213
query66 791 304 329 304
query67 16037 15755 15683 15683
query68 4470 547 558 547
query69 532 307 299 299
query70 1146 1081 1095 1081
query71 387 274 284 274
query72 7315 3943 3951 3943
query73 791 354 367 354
query74 10459 9013 9108 9013
query75 3486 2646 2687 2646
query76 3009 961 928 928
query77 584 306 296 296
query78 10547 9590 9555 9555
query79 2167 600 620 600
query80 937 450 475 450
query81 566 255 243 243
query82 715 144 143 143
query83 295 139 141 139
query84 276 70 70 70
query85 947 306 293 293
query86 337 309 287 287
query87 4817 4745 4682 4682
query88 3872 2244 2186 2186
query89 412 288 295 288
query90 1833 187 190 187
query91 137 104 100 100
query92 60 49 49 49
query93 1164 533 545 533
query94 804 288 281 281
query95 355 253 247 247
query96 624 279 286 279
query97 2883 2698 2736 2698
query98 210 212 191 191
query99 1606 1307 1329 1307
Total cold run time: 298472 ms
Total hot run time: 193111 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-3440747952
> > 1. A query_id is randomly generated to replace t_query_plan_info.query_id. external query does not need to report anything to FE, so the query_id can be changed.
> > Otherwise, multiple independent concurrent open tablet scanners have the same query_id. when one of the scanners ends, the other scanners will be canceled through FragmentMgr.cancel(query_id).
> > 2. (query_id, fragment_id) is executed only on one BE, locks _pipeline_map.
> > 3. External query (flink/spark read tablets) not need to report to FE.
>
> according to your describe，when multiple independent concurrent open tablet scanners ，must cause an error，why has this problem，I cannot understand it .
>
> https://doris-forum.org.cn/questions/D15j4/tong-guo-de-fang-shi-fang-wen-biao-ti-shi-bao-cuo
In the Spark/Flink connector, `openScanner` executes to read a tablet from Doris. After the read, `closeScanner` executes `FragmentMgr.cancel(query_id)` to clean up the remaining state in Doris. Therefore, if multiple openScanners use the same query_id, the issue mentioned in the PR may occur.
The actual error may occur randomly. because of the delay in executing `closeScanner -> FragmentMgr.cancel`, openScanners started at the same time may complete at the same time. Therefore, by the time the first completed scanner executes the cancel call, the other scanners have also completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2435087180
<details>
<summary>ClickBench: <b>Total hot run time: 31.7 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7846e1e11809abe0fda8e83193270c2e3c6e47f8, data reload: false
query1 0.03 0.03 0.03
query2 0.07 0.03 0.03
query3 0.23 0.07 0.07
query4 1.63 0.10 0.10
query5 0.42 0.39 0.40
query6 1.14 0.65 0.64
query7 0.02 0.02 0.02
query8 0.04 0.03 0.04
query9 0.57 0.50 0.49
query10 0.57 0.55 0.55
query11 0.15 0.12 0.12
query12 0.13 0.11 0.11
query13 0.60 0.60 0.59
query14 2.77 2.73 2.86
query15 0.89 0.82 0.85
query16 0.39 0.38 0.39
query17 1.10 1.06 1.06
query18 0.20 0.20 0.20
query19 1.96 1.83 2.04
query20 0.01 0.01 0.01
query21 15.38 0.59 0.59
query22 3.01 2.19 1.54
query23 17.06 1.11 0.89
query24 3.07 1.30 0.22
query25 0.23 0.17 0.15
query26 0.40 0.13 0.13
query27 0.05 0.05 0.04
query28 11.43 1.08 1.06
query29 12.58 3.31 3.33
query30 0.24 0.07 0.06
query31 2.88 0.39 0.38
query32 3.24 0.46 0.46
query33 3.01 3.08 3.02
query34 17.12 4.41 4.46
query35 4.53 4.51 4.44
query36 0.69 0.50 0.49
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.16 0.12 0.13
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 108.33 s
Total hot run time: 31.7 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2437175804
<details>
<summary>TPC-DS: <b>Total hot run time: 192627 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3075d09fbf919584fefcc795b3b3b8816c8ce87d, data reload: false
query1 865 410 414 410
query2 6283 2142 2059 2059
query3 8681 199 201 199
query4 35350 23975 23574 23574
query5 4904 451 447 447
query6 276 171 171 171
query7 4200 295 297 295
query8 293 232 240 232
query9 9440 2653 2654 2653
query10 501 260 252 252
query11 18138 15334 15298 15298
query12 158 106 109 106
query13 1575 421 432 421
query14 9820 7388 7352 7352
query15 256 170 183 170
query16 8042 505 513 505
query17 1431 586 579 579
query18 2242 315 321 315
query19 373 163 153 153
query20 129 114 118 114
query21 201 110 111 110
query22 4751 4608 4684 4608
query23 34744 34080 33999 33999
query24 11159 2816 2842 2816
query25 618 392 395 392
query26 1222 158 175 158
query27 2614 285 285 285
query28 7819 2437 2406 2406
query29 866 425 420 420
query30 262 172 168 168
query31 1029 820 787 787
query32 97 59 58 58
query33 764 281 285 281
query34 924 535 522 522
query35 1024 904 854 854
query36 1089 962 961 961
query37 128 72 74 72
query38 4585 4259 4358 4259
query39 1478 1435 1408 1408
query40 204 100 100 100
query41 52 53 51 51
query42 112 99 107 99
query43 545 497 503 497
query44 1286 827 824 824
query45 181 167 170 167
query46 1146 712 713 712
query47 1965 1909 1863 1863
query48 435 319 336 319
query49 921 427 406 406
query50 817 392 410 392
query51 7043 6942 6997 6942
query52 104 91 91 91
query53 263 183 189 183
query54 1094 407 415 407
query55 76 79 77 77
query56 272 253 275 253
query57 1335 1176 1173 1173
query58 215 201 210 201
query59 3247 2964 2895 2895
query60 277 248 255 248
query61 103 106 111 106
query62 864 681 667 667
query63 222 191 182 182
query64 4058 630 610 610
query65 3281 3242 3264 3242
query66 898 314 306 306
query67 16014 15708 15616 15616
query68 4395 541 567 541
query69 466 254 253 253
query70 1218 1174 1138 1138
query71 415 283 280 280
query72 6274 3942 3945 3942
query73 776 362 361 361
query74 10314 9031 9030 9030
query75 3458 2669 2698 2669
query76 2808 1080 1030 1030
query77 425 284 295 284
query78 10534 9587 9492 9492
query79 1363 588 604 588
query80 967 426 438 426
query81 569 241 249 241
query82 663 116 118 116
query83 243 139 139 139
query84 241 69 71 69
query85 1377 290 282 282
query86 390 299 290 290
query87 4813 4610 4607 4607
query88 3540 2252 2202 2202
query89 404 288 295 288
query90 1967 186 192 186
query91 141 114 112 112
query92 59 50 51 50
query93 1482 536 534 534
query94 917 295 270 270
query95 343 251 255 251
query96 623 292 282 282
query97 2868 2706 2662 2662
query98 211 200 200 200
query99 1585 1323 1314 1314
Total cold run time: 301210 ms
Total hot run time: 192627 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2437097594
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](flink) Fix scanner be cancelled early on pipelineX [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42421:
URL: https://github.com/apache/doris/pull/42421#issuecomment-2437151065
<details>
<summary>TPC-H: <b>Total hot run time: 41786 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3075d09fbf919584fefcc795b3b3b8816c8ce87d, data reload: false
------ Round 1 ----------------------------------
q1 17581 8121 7337 7337
q2 2034 282 278 278
q3 12061 1107 1146 1107
q4 10574 899 885 885
q5 7752 3084 3080 3080
q6 239 155 154 154
q7 1040 618 594 594
q8 9369 1990 2028 1990
q9 7688 6424 6415 6415
q10 7044 2457 2465 2457
q11 444 256 255 255
q12 417 225 229 225
q13 17791 3026 3058 3026
q14 241 204 211 204
q15 568 505 535 505
q16 657 587 580 580
q17 976 589 588 588
q18 7346 6844 6721 6721
q19 1325 1021 1072 1021
q20 487 185 190 185
q21 4016 3238 3157 3157
q22 1120 1022 1035 1022
Total cold run time: 110770 ms
Total hot run time: 41786 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7321 7291 7365 7291
q2 339 236 235 235
q3 3080 2964 3073 2964
q4 2084 1880 1781 1781
q5 5736 5776 5802 5776
q6 233 145 145 145
q7 2254 1920 1845 1845
q8 3459 3492 3565 3492
q9 9083 8984 8865 8865
q10 3606 3559 3579 3559
q11 601 509 493 493
q12 828 599 613 599
q13 9105 3189 3257 3189
q14 305 296 296 296
q15 587 523 521 521
q16 690 626 637 626
q17 1850 1645 1638 1638
q18 8253 7735 7653 7653
q19 1710 1589 1615 1589
q20 2127 1836 1855 1836
q21 5673 5562 5468 5468
q22 1134 1073 1067 1067
Total cold run time: 70058 ms
Total hot run time: 60928 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org