You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yiguolei (via GitHub)" <gi...@apache.org> on 2024/01/12 06:46:31 UTC
[PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
yiguolei opened a new pull request, #29912:
URL: https://github.com/apache/doris/pull/29912
## Proposed changes
serving_blocks_num is an atomic variable. It's ++ and -- are not protected by transfer lock. I think it maybe the root cause of query timeout. So that I remove the check and test it in github pipeline.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1888599563
<details>
<summary>ClickBench: <b>Total hot run time: 31.11 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b34f111f8adad9bad576e6c91067b36fbe4c6eef, data reload: false
query1 0.06 0.06 0.05
query2 0.06 0.02 0.02
query3 0.25 0.12 0.13
query4 1.76 0.13 0.13
query5 0.53 0.53 0.52
query6 1.35 0.64 0.62
query7 0.02 0.01 0.01
query8 0.05 0.03 0.02
query9 0.55 0.49 0.50
query10 0.57 0.55 0.57
query11 0.11 0.09 0.09
query12 0.11 0.09 0.09
query13 0.60 0.60 0.60
query14 0.77 0.77 0.79
query15 0.80 0.78 0.77
query16 0.36 0.39 0.40
query17 1.01 0.99 1.02
query18 0.24 0.26 0.24
query19 1.86 1.77 1.78
query20 0.02 0.01 0.00
query21 15.41 0.56 0.55
query22 2.56 2.45 2.27
query23 17.36 0.75 0.76
query24 16.03 0.59 0.57
query25 2.16 0.15 0.15
query26 0.13 0.14 0.12
query27 0.15 0.15 0.14
query28 6.19 0.81 0.80
query29 12.59 3.19 3.20
query30 0.61 0.52 0.48
query31 2.78 0.34 0.35
query32 3.40 0.48 0.47
query33 3.21 3.23 3.27
query34 15.94 4.28 4.22
query35 4.24 4.28 4.25
query36 1.11 1.09 1.08
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.01 0.01
query40 0.16 0.13 0.13
query41 0.08 0.01 0.02
query42 0.02 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 115.36 s
Total hot run time: 31.11 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #29912:
URL: https://github.com/apache/doris/pull/29912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1888584982
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.22 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.7 seconds inserted 10000000 Rows, about 460K ops/s
storage size: 17183387192 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1888603832
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit b34f111f8adad9bad576e6c91067b36fbe4c6eef with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1888520621
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1888581702
<details>
<summary>TPC-DS: <b>Total hot run time: 177856 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b34f111f8adad9bad576e6c91067b36fbe4c6eef, data reload: false
query1 932 337 328 328
query2 6725 1877 1799 1799
query3 6721 209 200 200
query4 32078 22146 21998 21998
query5 6954 565 587 565
query6 269 185 206 185
query7 4616 275 282 275
query8 252 200 187 187
query9 9144 2776 2831 2776
query10 484 245 247 245
query11 16284 15324 15259 15259
query12 127 71 69 69
query13 1701 394 371 371
query14 11203 7455 7437 7437
query15 212 177 183 177
query16 4623 255 251 251
query17 1001 489 478 478
query18 1799 257 262 257
query19 182 161 153 153
query20 77 81 72 72
query21 191 123 122 122
query22 4970 4716 4633 4633
query23 32061 31079 30762 30762
query24 11452 2828 2760 2760
query25 576 324 328 324
query26 1627 154 152 152
query27 3239 287 277 277
query28 7496 1868 1854 1854
query29 1545 701 686 686
query30 288 140 143 140
query31 937 749 769 749
query32 87 61 66 61
query33 697 245 248 245
query34 1141 478 473 473
query35 902 799 781 781
query36 1374 1263 1269 1263
query37 101 66 71 66
query38 3434 3266 3296 3266
query39 1320 1263 1260 1260
query40 219 99 100 99
query41 38 34 33 33
query42 101 95 93 93
query43 533 495 465 465
query44 1091 702 720 702
query45 201 183 184 183
query46 1049 636 642 636
query47 1655 1560 1567 1560
query48 403 324 312 312
query49 1162 316 323 316
query50 691 319 317 317
query51 5365 5176 5219 5176
query52 97 89 98 89
query53 364 290 285 285
query54 910 451 462 451
query55 94 88 88 88
query56 212 186 192 186
query57 1000 945 915 915
query58 218 190 195 190
query59 2882 2634 2704 2634
query60 236 230 214 214
query61 84 84 87 84
query62 661 395 381 381
query63 318 296 292 292
query64 5124 1655 1656 1655
query65 3363 3258 3275 3258
query66 1291 330 327 327
query67 15588 15057 14863 14863
query68 13637 502 503 502
query69 632 330 322 322
query70 2165 1506 1520 1506
query71 504 238 258 238
query72 5377 2878 2821 2821
query73 3786 314 321 314
query74 6989 6420 6406 6406
query75 5076 2337 2336 2336
query76 5908 999 1043 999
query77 732 268 290 268
query78 9270 8665 8634 8634
query79 2180 502 499 499
query80 572 347 345 345
query81 470 200 206 200
query82 210 88 92 88
query83 167 121 128 121
query84 284 74 72 72
query85 1075 345 333 333
query86 391 410 413 410
query87 3600 3374 3382 3374
query88 2817 2176 2163 2163
query89 450 395 386 386
query90 1998 212 192 192
query91 167 133 122 122
query92 68 52 51 51
query93 1308 439 426 426
query94 1271 179 173 173
query95 528 477 457 457
query96 619 316 321 316
query97 4292 4133 4153 4133
query98 209 192 184 184
query99 1001 733 705 705
Total cold run time: 301525 ms
Total hot run time: 177856 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1889332116
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1888527205
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1888569940
<details>
<summary>TPC-H: <b>Total hot run time: 39997 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b34f111f8adad9bad576e6c91067b36fbe4c6eef, data reload: false
------ Round 1 ----------------------------------
q1 19250 6473 5956 5956
q2 2449 194 166 166
q3 11727 1175 1205 1175
q4 10259 801 768 768
q5 7825 3200 3183 3183
q6 207 123 126 123
q7 904 496 484 484
q8 9235 2041 2043 2041
q9 7367 6405 6416 6405
q10 8236 3061 3083 3061
q11 422 213 213 213
q12 365 194 195 194
q13 18073 3435 3393 3393
q14 251 214 210 210
q15 564 516 528 516
q16 450 378 376 376
q17 949 572 479 479
q18 7706 7030 6918 6918
q19 1618 1299 1415 1299
q20 585 312 288 288
q21 2778 2482 2429 2429
q22 396 320 322 320
Total cold run time: 111616 ms
Total hot run time: 39997 ms
----- Round 2, with runtime_filter_mode=off -----
q1 6007 5455 5698 5455
q2 344 229 215 215
q3 3295 3100 3140 3100
q4 2234 2035 1899 1899
q5 6159 5850 5922 5850
q6 203 128 122 122
q7 2329 1963 1961 1961
q8 3716 3628 3675 3628
q9 9675 21208 9479 9479
q10 5688 3673 3826 3673
q11 646 473 445 445
q12 1140 667 646 646
q13 17155 3276 3240 3240
q14 353 268 272 268
q15 571 517 506 506
q16 525 485 467 467
q17 9328 1941 1856 1856
q18 9390 9538 12037 9538
q19 13090 1570 1604 1570
q20 3811 1941 1929 1929
q21 14136 5677 5887 5677
q22 1016 614 598 598
Total cold run time: 110811 ms
Total hot run time: 62122 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1888539475
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1888532643
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1888572968
TeamCity be ut coverage result:
Function Coverage: 36.72% (8652/23565)
Line Coverage: 28.74% (70489/245282)
Region Coverage: 27.65% (36420/131713)
Branch Coverage: 24.37% (18640/76472)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b34f111f8adad9bad576e6c91067b36fbe4c6eef_b34f111f8adad9bad576e6c91067b36fbe4c6eef/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29912:
URL: https://github.com/apache/doris/pull/29912#issuecomment-1889332227
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org