You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "eldenmoon (via GitHub)" <gi...@apache.org> on 2024/01/11 10:25:32 UTC
[PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
eldenmoon opened a new pull request, #29872:
URL: https://github.com/apache/doris/pull/29872
…ated row entry
1. Add more info to trace potential bug and avoid crash
2. use correct permutation size to do `column->permute`
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29872:
URL: https://github.com/apache/doris/pull/29872#issuecomment-1886851087
TeamCity be ut coverage result:
Function Coverage: 36.70% (8647/23560)
Line Coverage: 28.73% (70441/245172)
Region Coverage: 27.64% (36388/131635)
Branch Coverage: 24.36% (18625/76456)
Coverage Report: http://coverage.selectdb-in.cc/coverage/97be709eb7288dd613d83428cec00c9613710a1b_97be709eb7288dd613d83428cec00c9613710a1b/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29872:
URL: https://github.com/apache/doris/pull/29872#issuecomment-1891383348
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #29872:
URL: https://github.com/apache/doris/pull/29872#issuecomment-1886806011
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29872:
URL: https://github.com/apache/doris/pull/29872#issuecomment-1886881556
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.41 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s
storage size: 17184213712 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29872:
URL: https://github.com/apache/doris/pull/29872#issuecomment-1886835673
<details>
<summary>TPC-H: <b>Total hot run time: 39093 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97be709eb7288dd613d83428cec00c9613710a1b, data reload: false
------ Round 1 ----------------------------------
q1 17957 5099 5023 5023
q2 2030 138 134 134
q3 10706 1168 1168 1168
q4 10558 802 850 802
q5 8110 3294 3270 3270
q6 211 128 130 128
q7 922 548 510 510
q8 9812 2021 2041 2021
q9 7964 6560 6529 6529
q10 8268 3095 3069 3069
q11 417 218 199 199
q12 371 194 194 194
q13 18107 3428 3449 3428
q14 237 221 240 221
q15 557 522 513 513
q16 434 379 380 379
q17 948 492 557 492
q18 7395 6741 6723 6723
q19 1586 1358 1320 1320
q20 528 332 283 283
q21 2783 2385 2417 2385
q22 351 305 302 302
Total cold run time: 110252 ms
Total hot run time: 39093 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5044 4961 4963 4961
q2 307 198 205 198
q3 3359 3328 3312 3312
q4 2245 2258 2250 2250
q5 5830 5814 5842 5814
q6 195 117 120 117
q7 2321 1890 1869 1869
q8 3458 3569 3600 3569
q9 8903 8871 8836 8836
q10 3759 3853 3833 3833
q11 559 458 422 422
q12 785 612 618 612
q13 4048 3241 3230 3230
q14 279 257 267 257
q15 591 508 507 507
q16 513 455 443 443
q17 2075 2021 2045 2021
q18 8774 8334 8374 8334
q19 1619 1608 1610 1608
q20 2159 1947 1932 1932
q21 6121 5640 5714 5640
q22 544 467 474 467
Total cold run time: 63488 ms
Total hot run time: 60232 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29872:
URL: https://github.com/apache/doris/pull/29872#issuecomment-1886887605
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 97be709eb7288dd613d83428cec00c9613710a1b with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 12.7 seconds inserted 10000000 Rows, about 787K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29872:
URL: https://github.com/apache/doris/pull/29872#issuecomment-1886880983
<details>
<summary>ClickBench: <b>Total hot run time: 30.46 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97be709eb7288dd613d83428cec00c9613710a1b, data reload: false
query1 0.06 0.06 0.05
query2 0.06 0.02 0.02
query3 0.26 0.11 0.11
query4 1.76 0.13 0.12
query5 0.52 0.52 0.52
query6 1.39 0.61 0.63
query7 0.01 0.01 0.01
query8 0.04 0.02 0.02
query9 0.55 0.49 0.50
query10 0.55 0.54 0.55
query11 0.12 0.08 0.09
query12 0.13 0.10 0.09
query13 0.61 0.60 0.59
query14 0.78 0.79 0.76
query15 0.81 0.81 0.80
query16 0.36 0.35 0.36
query17 0.99 1.00 0.99
query18 0.27 0.27 0.25
query19 1.77 1.78 1.80
query20 0.01 0.01 0.01
query21 15.42 0.57 0.59
query22 2.81 3.01 1.80
query23 17.28 0.93 0.76
query24 16.14 0.57 0.59
query25 2.15 0.15 0.15
query26 0.14 0.13 0.13
query27 0.15 0.15 0.14
query28 5.95 0.82 0.82
query29 12.52 3.24 3.16
query30 0.52 0.50 0.50
query31 2.78 0.34 0.36
query32 3.46 0.49 0.48
query33 3.21 3.27 3.24
query34 15.89 4.16 4.16
query35 4.19 4.19 4.17
query36 1.11 1.04 1.06
query37 0.06 0.04 0.05
query38 0.05 0.02 0.03
query39 0.02 0.01 0.02
query40 0.17 0.13 0.14
query41 0.06 0.01 0.02
query42 0.02 0.01 0.01
query43 0.02 0.02 0.02
Total cold run time: 115.17 s
Total hot run time: 30.46 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29872:
URL: https://github.com/apache/doris/pull/29872#issuecomment-1886816734
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29872:
URL: https://github.com/apache/doris/pull/29872#issuecomment-1891383390
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye merged PR #29872:
URL: https://github.com/apache/doris/pull/29872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](topn opt) avoid crash when rpc returned row contains duplic… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29872:
URL: https://github.com/apache/doris/pull/29872#issuecomment-1886868439
<details>
<summary>TPC-DS: <b>Total hot run time: 180156 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97be709eb7288dd613d83428cec00c9613710a1b, data reload: false
query1 921 339 337 337
query2 6703 1897 1890 1890
query3 6711 219 207 207
query4 26306 22160 22311 22160
query5 5960 527 515 515
query6 277 190 185 185
query7 4612 280 286 280
query8 235 209 205 205
query9 9153 2751 2924 2751
query10 496 248 245 245
query11 16197 15409 15517 15409
query12 135 76 73 73
query13 1746 398 373 373
query14 13105 7458 7221 7221
query15 269 196 199 196
query16 6477 253 245 245
query17 1950 494 475 475
query18 1944 271 273 271
query19 284 162 149 149
query20 83 78 74 74
query21 199 132 126 126
query22 4999 4828 4803 4803
query23 31950 31014 31058 31014
query24 11985 2863 2827 2827
query25 553 329 330 329
query26 1745 153 154 153
query27 2924 270 288 270
query28 7109 1855 1858 1855
query29 2059 699 682 682
query30 280 148 140 140
query31 958 768 777 768
query32 94 60 63 60
query33 718 249 261 249
query34 1106 482 471 471
query35 862 725 727 725
query36 1385 1246 1246 1246
query37 185 72 70 70
query38 3428 3289 3261 3261
query39 1349 1287 1274 1274
query40 319 108 101 101
query41 39 36 36 36
query42 104 94 96 94
query43 559 523 503 503
query44 1136 722 724 722
query45 212 191 190 190
query46 1046 668 660 660
query47 1706 1591 1579 1579
query48 404 325 315 315
query49 1261 335 312 312
query50 708 327 323 323
query51 5348 5224 5212 5212
query52 102 101 95 95
query53 364 300 306 300
query54 2048 1616 1625 1616
query55 90 90 90 90
query56 220 207 206 206
query57 1056 988 975 975
query58 220 199 195 195
query59 2837 2563 2642 2563
query60 247 223 241 223
query61 84 80 81 80
query62 627 413 394 394
query63 315 296 291 291
query64 5722 1679 1656 1656
query65 3374 3302 3287 3287
query66 1283 344 332 332
query67 15667 15461 15085 15085
query68 12829 556 540 540
query69 648 386 387 386
query70 1677 1507 1555 1507
query71 509 228 241 228
query72 5029 2815 2801 2801
query73 3125 314 328 314
query74 6976 6503 6438 6438
query75 5470 2306 2349 2306
query76 6321 922 1143 922
query77 930 272 292 272
query78 10080 8540 8502 8502
query79 1049 520 491 491
query80 628 336 362 336
query81 456 209 217 209
query82 208 89 95 89
query83 201 123 121 121
query84 285 76 75 75
query85 1056 349 329 329
query86 376 405 367 367
query87 3580 3409 3399 3399
query88 2903 2253 2237 2237
query89 452 399 384 384
query90 1994 222 210 210
query91 159 123 123 123
query92 66 55 55 55
query93 1248 464 455 455
query94 1227 188 184 184
query95 527 481 473 473
query96 628 326 313 313
query97 4279 4163 4144 4144
query98 200 188 191 188
query99 1014 739 742 739
Total cold run time: 300680 ms
Total hot run time: 180156 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org