You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "qidaye (via GitHub)" <gi...@apache.org> on 2024/03/07 09:47:36 UTC
[PR] [fix](inverted index)revert compare_postings optimization in clucene … [doris]
qidaye opened a new pull request, #31948:
URL: https://github.com/apache/doris/pull/31948
…to avoid load core
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index)revert compare_postings optimization in clucene … [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31948:
URL: https://github.com/apache/doris/pull/31948#issuecomment-1983190425
<details>
<summary>TPC-H: <b>Total hot run time: 49683 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 566af1d1b9a57042286f699d6147d9ec989ecc10, data reload: false
------ Round 1 ----------------------------------
q1 17706 4367 4339 4339
q2 2056 146 140 140
q3 10368 1855 1908 1855
q4 10366 1208 1287 1208
q5 8638 3882 3949 3882
q6 226 121 122 121
q7 2014 1580 1579 1579
q8 9285 2686 2709 2686
q9 10825 10472 10481 10472
q10 8624 3501 3491 3491
q11 429 243 246 243
q12 466 302 299 299
q13 18345 3949 4025 3949
q14 353 333 315 315
q15 501 463 462 462
q16 696 587 581 581
q17 1106 946 958 946
q18 7335 6773 6917 6773
q19 1661 1578 1551 1551
q20 504 324 302 302
q21 4481 4091 4101 4091
q22 501 398 401 398
Total cold run time: 116486 ms
Total hot run time: 49683 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4324 4253 4269 4253
q2 316 221 217 217
q3 4194 4166 4136 4136
q4 2759 2728 2735 2728
q5 7265 7212 7258 7212
q6 230 122 117 117
q7 3195 2838 2817 2817
q8 4372 4490 4525 4490
q9 17143 16845 16895 16845
q10 4203 4249 4289 4249
q11 744 704 702 702
q12 1027 862 846 846
q13 6589 3690 3744 3690
q14 447 425 415 415
q15 506 453 457 453
q16 769 708 711 708
q17 3894 3900 3906 3900
q18 8878 8682 8678 8678
q19 1702 1703 1655 1655
q20 2395 2159 2136 2136
q21 8551 8492 8458 8458
q22 1059 953 985 953
Total cold run time: 84562 ms
Total hot run time: 79658 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index)revert compare_postings optimization in clucene … [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31948:
URL: https://github.com/apache/doris/pull/31948#issuecomment-1983138426
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index)revert compare_postings optimization in clucene … [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31948:
URL: https://github.com/apache/doris/pull/31948#issuecomment-1983212413
<details>
<summary>TPC-DS: <b>Total hot run time: 202842 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 566af1d1b9a57042286f699d6147d9ec989ecc10, data reload: false
query1 914 387 383 383
query2 6530 2322 2153 2153
query3 6906 196 197 196
query4 19979 17984 18026 17984
query5 19709 6483 6518 6483
query6 298 221 226 221
query7 4152 298 306 298
query8 253 248 239 239
query9 3108 2671 2589 2589
query10 410 278 300 278
query11 11330 10700 10761 10700
query12 130 68 70 68
query13 5570 639 629 629
query14 17532 13660 13945 13660
query15 362 231 229 229
query16 6447 277 278 277
query17 1700 1466 869 869
query18 2308 406 398 398
query19 213 147 147 147
query20 74 77 78 77
query21 183 89 92 89
query22 5370 4972 5150 4972
query23 32396 31889 31871 31871
query24 7063 6484 6516 6484
query25 521 430 428 428
query26 530 159 156 156
query27 1906 297 290 290
query28 6102 2229 2186 2186
query29 2950 2872 2850 2850
query30 242 161 163 161
query31 904 705 717 705
query32 69 62 53 53
query33 393 248 249 248
query34 844 481 468 468
query35 1100 905 940 905
query36 1867 1510 1531 1510
query37 87 62 58 58
query38 3055 2971 2949 2949
query39 1379 1318 1321 1318
query40 200 96 98 96
query41 35 33 32 32
query42 96 88 85 85
query43 619 635 719 635
query44 1117 718 707 707
query45 235 227 226 226
query46 1224 987 960 960
query47 1960 1882 1917 1882
query48 981 694 664 664
query49 611 379 373 373
query50 868 674 587 587
query51 4747 4613 4692 4613
query52 88 85 87 85
query53 440 322 325 322
query54 2663 2441 2437 2437
query55 86 77 77 77
query56 218 208 207 207
query57 1130 1030 1031 1030
query58 204 211 207 207
query59 3549 3225 3363 3225
query60 205 195 188 188
query61 83 80 83 80
query62 822 492 476 476
query63 497 337 341 337
query64 2316 1503 1369 1369
query65 3643 3567 3604 3567
query66 758 368 377 368
query67 18125 15726 15884 15726
query68 8558 655 663 655
query69 558 345 350 345
query70 1991 1654 1867 1654
query71 403 290 302 290
query72 6383 3455 3416 3416
query73 731 326 324 324
query74 6302 5791 5933 5791
query75 4752 3737 3693 3693
query76 4892 1160 1224 1160
query77 675 254 251 251
query78 12504 11407 11674 11407
query79 7560 632 650 632
query80 1011 391 380 380
query81 502 234 240 234
query82 1495 105 95 95
query83 159 129 132 129
query84 260 68 71 68
query85 845 284 274 274
query86 393 405 373 373
query87 3223 3006 3037 3006
query88 5133 2306 2321 2306
query89 355 273 292 273
query90 1917 212 208 208
query91 155 116 117 116
query92 56 49 53 49
query93 4600 603 579 579
query94 739 202 209 202
query95 1101 1073 1093 1073
query96 648 324 327 324
query97 6502 6312 6322 6312
query98 199 179 173 173
query99 2966 987 910 910
Total cold run time: 309974 ms
Total hot run time: 202842 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index)revert compare_postings optimization in clucene … [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31948:
URL: https://github.com/apache/doris/pull/31948#issuecomment-1983221498
<details>
<summary>ClickBench: <b>Total hot run time: 31.13 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 566af1d1b9a57042286f699d6147d9ec989ecc10, data reload: false
query1 0.02 0.02 0.02
query2 0.06 0.03 0.02
query3 0.24 0.05 0.05
query4 1.83 0.07 0.06
query5 0.52 0.52 0.52
query6 1.24 0.62 0.63
query7 0.01 0.01 0.00
query8 0.03 0.02 0.03
query9 0.54 0.48 0.48
query10 0.54 0.53 0.54
query11 0.12 0.09 0.09
query12 0.12 0.09 0.09
query13 0.62 0.61 0.61
query14 0.79 0.80 0.78
query15 0.78 0.76 0.77
query16 0.38 0.36 0.36
query17 1.01 1.01 1.00
query18 0.24 0.25 0.26
query19 1.89 1.79 1.85
query20 0.02 0.01 0.01
query21 15.47 0.55 0.55
query22 1.69 1.76 2.15
query23 17.12 0.92 1.15
query24 3.82 1.17 4.21
query25 0.42 0.10 0.04
query26 0.72 0.16 0.15
query27 0.05 0.04 0.04
query28 6.15 0.72 0.74
query29 12.62 2.35 2.32
query30 0.57 0.52 0.53
query31 2.82 0.38 0.39
query32 3.39 0.50 0.50
query33 3.06 3.11 3.08
query34 15.30 4.81 4.81
query35 4.89 4.86 4.85
query36 1.06 1.01 1.01
query37 0.06 0.05 0.04
query38 0.03 0.02 0.02
query39 0.02 0.01 0.02
query40 0.16 0.14 0.14
query41 0.06 0.02 0.01
query42 0.02 0.01 0.01
query43 0.02 0.02 0.02
Total cold run time: 100.52 s
Total hot run time: 31.13 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index)revert compare_postings optimization in clucene to avoid load core [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang merged PR #31948:
URL: https://github.com/apache/doris/pull/31948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index)revert compare_postings optimization in clucene … [doris]
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye commented on PR #31948:
URL: https://github.com/apache/doris/pull/31948#issuecomment-1983128680
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index)revert compare_postings optimization in clucene … [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31948:
URL: https://github.com/apache/doris/pull/31948#issuecomment-1983227008
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 566af1d1b9a57042286f699d6147d9ec989ecc10 with default session variables
Stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 20.8 seconds inserted 10000000 Rows, about 480K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index)revert compare_postings optimization in clucene … [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31948:
URL: https://github.com/apache/doris/pull/31948#issuecomment-1983126191
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index)revert compare_postings optimization in clucene … [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31948:
URL: https://github.com/apache/doris/pull/31948#issuecomment-1983180601
TeamCity be ut coverage result:
Function Coverage: 37.80% (8037/21263)
Line Coverage: 29.48% (65572/222421)
Region Coverage: 28.95% (33745/116569)
Branch Coverage: 24.79% (17313/69846)
Coverage Report: http://coverage.selectdb-in.cc/coverage/566af1d1b9a57042286f699d6147d9ec989ecc10_566af1d1b9a57042286f699d6147d9ec989ecc10/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org