You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HappenLee (via GitHub)" <gi...@apache.org> on 2024/03/05 11:51:57 UTC
[PR] [Performance](sink) opt mysql result writer [doris]
HappenLee opened a new pull request, #31816:
URL: https://github.com/apache/doris/pull/31816
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-2026612247
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-2024664772
<details>
<summary>TPC-DS: <b>Total hot run time: 181571 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3dc6ddb8ed73eaeb018ea804e57a7878276bd24, data reload: false
query1 954 362 350 350
query2 6538 1888 1872 1872
query3 6705 217 210 210
query4 31669 21483 21435 21435
query5 4316 397 390 390
query6 262 175 192 175
query7 4635 298 289 289
query8 235 172 174 172
query9 9134 2339 2319 2319
query10 574 252 249 249
query11 15490 14224 14172 14172
query12 134 90 86 86
query13 1619 424 403 403
query14 9576 7651 8039 7651
query15 263 186 180 180
query16 8231 261 263 261
query17 1987 578 559 559
query18 2106 294 286 286
query19 343 156 171 156
query20 92 92 90 90
query21 208 134 126 126
query22 5017 4869 4772 4772
query23 33712 32720 32695 32695
query24 12087 2881 2918 2881
query25 652 395 368 368
query26 1788 154 151 151
query27 3103 350 356 350
query28 7683 1892 1879 1879
query29 1027 623 604 604
query30 305 145 147 145
query31 1000 714 738 714
query32 93 56 56 56
query33 763 241 244 241
query34 1147 490 486 486
query35 835 615 599 599
query36 1016 906 894 894
query37 273 65 63 63
query38 3551 3423 3438 3423
query39 1477 1459 1405 1405
query40 294 110 113 110
query41 49 46 45 45
query42 102 98 96 96
query43 472 438 431 431
query44 1176 731 734 731
query45 281 255 251 251
query46 1101 695 685 685
query47 1926 1845 1849 1845
query48 447 360 354 354
query49 1233 327 333 327
query50 753 373 369 369
query51 6840 6656 6618 6618
query52 110 89 91 89
query53 354 281 273 273
query54 311 240 232 232
query55 89 79 84 79
query56 243 224 218 218
query57 1217 1126 1141 1126
query58 221 201 205 201
query59 2778 2688 2565 2565
query60 256 245 245 245
query61 107 115 113 113
query62 643 425 432 425
query63 314 277 288 277
query64 7007 4151 3996 3996
query65 3158 3021 3027 3021
query66 1471 399 369 369
query67 15612 14871 15071 14871
query68 5664 522 523 522
query69 599 373 382 373
query70 1207 1233 1217 1217
query71 457 271 265 265
query72 6410 2704 2508 2508
query73 736 322 325 322
query74 7783 6575 6355 6355
query75 3179 2206 2192 2192
query76 4242 873 888 873
query77 607 262 256 256
query78 11035 10190 10227 10190
query79 8476 539 528 528
query80 1618 378 378 378
query81 505 221 216 216
query82 1462 85 88 85
query83 199 144 140 140
query84 281 80 73 73
query85 1518 332 315 315
query86 469 298 291 291
query87 3789 3523 3535 3523
query88 5113 2300 2299 2299
query89 512 367 374 367
query90 1995 177 175 175
query91 176 139 136 136
query92 57 49 49 49
query93 7083 497 485 485
query94 1171 180 173 173
query95 427 336 333 333
query96 622 272 268 268
query97 2689 2514 2500 2500
query98 235 214 211 211
query99 1240 853 834 834
Total cold run time: 310721 ms
Total hot run time: 181571 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-2026612140
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-2024644557
TeamCity be ut coverage result:
Function Coverage: 35.63% (8835/24797)
Line Coverage: 27.34% (72388/264734)
Region Coverage: 26.56% (37531/141291)
Branch Coverage: 23.37% (19141/81906)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d3dc6ddb8ed73eaeb018ea804e57a7878276bd24_d3dc6ddb8ed73eaeb018ea804e57a7878276bd24/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-2024676350
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit d3dc6ddb8ed73eaeb018ea804e57a7878276bd24 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.7 seconds inserted 10000000 Rows, about 729K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee merged PR #31816:
URL: https://github.com/apache/doris/pull/31816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-1978587065
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-1979297074
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-2024622118
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-2024622610
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-1978586896
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-1979457677
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31816:
URL: https://github.com/apache/doris/pull/31816#discussion_r1512705310
##########
be/src/util/url_coding.cpp:
##########
@@ -21,6 +21,7 @@
#include <memory>
#include <sstream>
+#include <turbob64.h>
Review Comment:
warning: 'turbob64.h' file not found [clang-diagnostic-error]
```cpp
#include <turbob64.h>
^
```
##########
be/src/util/url_coding.cpp:
##########
@@ -91,7 +92,7 @@
size_t len = in.size();
// Every 3 source bytes will be encoded into 4 bytes.
std::unique_ptr<unsigned char[]> buf(new unsigned char[(((len + 2) / 3) * 4)]);
- const auto* s = reinterpret_cast<const unsigned char*>(in.data());
+ const unsigned char* s = reinterpret_cast<const unsigned char*>(in.data());
Review Comment:
warning: use auto when initializing with a cast to avoid duplicating the type name [modernize-use-auto]
```suggestion
const auto* s = reinterpret_cast<const unsigned char*>(in.data());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](sink) opt mysql result writer [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31816:
URL: https://github.com/apache/doris/pull/31816#issuecomment-2024647093
<details>
<summary>TPC-H: <b>Total hot run time: 37773 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3dc6ddb8ed73eaeb018ea804e57a7878276bd24, data reload: false
------ Round 1 ----------------------------------
q1 18342 4411 4279 4279
q2 2556 168 160 160
q3 11704 1129 1240 1129
q4 10468 806 750 750
q5 7766 2970 2899 2899
q6 199 125 125 125
q7 1038 603 574 574
q8 9343 1984 1994 1984
q9 6912 6336 6325 6325
q10 8441 3484 3589 3484
q11 433 223 214 214
q12 365 210 195 195
q13 17802 2847 2891 2847
q14 242 208 204 204
q15 503 475 455 455
q16 493 367 372 367
q17 947 611 585 585
q18 7189 6506 6414 6414
q19 2975 1438 1472 1438
q20 546 258 249 249
q21 3569 2879 2812 2812
q22 339 284 294 284
Total cold run time: 112172 ms
Total hot run time: 37773 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4151 4073 4080 4073
q2 337 232 246 232
q3 2942 2827 2836 2827
q4 1831 1594 1570 1570
q5 5232 5207 5233 5207
q6 192 114 115 114
q7 2155 1782 1832 1782
q8 3163 3303 3267 3267
q9 8436 8400 8427 8400
q10 3759 3755 3754 3754
q11 538 444 444 444
q12 721 531 537 531
q13 16922 2894 2867 2867
q14 289 266 244 244
q15 499 458 459 458
q16 465 422 410 410
q17 1712 1501 1476 1476
q18 7499 7125 7063 7063
q19 1598 1559 1508 1508
q20 1923 1711 1696 1696
q21 4759 4632 4741 4632
q22 521 450 430 430
Total cold run time: 69644 ms
Total hot run time: 52985 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org