You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaijchen (via GitHub)" <gi...@apache.org> on 2023/12/05 12:54:16 UTC
[PR] [fix](move-memtable) temporary disable load stream profile [doris]
kaijchen opened a new pull request, #28029:
URL: https://github.com/apache/doris/pull/28029
## Proposed changes
Temporary disable load stream profile, because:
1. LoadStream profile is not processed by the upstream.
2. There are some problems in `_profile->to_thrift`.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28029:
URL: https://github.com/apache/doris/pull/28029#issuecomment-1841899087
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit de478d72f19beb95535aef4fb45e118403ea9f47, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4701 4415 4447 4415
q2 362 152 158 152
q3 1463 1233 1186 1186
q4 1115 909 862 862
q5 3175 3203 3228 3203
q6 249 131 129 129
q7 1024 507 489 489
q8 2226 2255 2206 2206
q9 6681 6675 6659 6659
q10 3237 3244 3271 3244
q11 328 205 207 205
q12 359 215 208 208
q13 4817 3853 3815 3815
q14 249 212 221 212
q15 567 523 531 523
q16 440 388 389 388
q17 1019 597 595 595
q18 7429 7310 7219 7219
q19 1528 1383 1401 1383
q20 572 376 296 296
q21 3091 2705 2744 2705
q22 358 291 300 291
Total cold run time: 44990 ms
Total hot run time: 40385 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4377 4396 4376 4376
q2 266 164 169 164
q3 3537 3529 3513 3513
q4 2386 2377 2361 2361
q5 5740 5731 5755 5731
q6 239 120 125 120
q7 2388 1832 1865 1832
q8 3526 3534 3531 3531
q9 9072 9015 9066 9015
q10 3909 4017 4026 4017
q11 524 393 373 373
q12 759 585 592 585
q13 4283 3546 3576 3546
q14 277 258 263 258
q15 569 520 517 517
q16 501 440 468 440
q17 1879 1854 1855 1854
q18 8613 8254 8178 8178
q19 1763 1774 1755 1755
q20 2256 2054 1944 1944
q21 6517 6205 6142 6142
q22 503 419 414 414
Total cold run time: 63884 ms
Total hot run time: 60666 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28029:
URL: https://github.com/apache/doris/pull/28029#issuecomment-1840750918
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #28029:
URL: https://github.com/apache/doris/pull/28029#issuecomment-1841866596
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28029:
URL: https://github.com/apache/doris/pull/28029#issuecomment-1840941565
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit de478d72f19beb95535aef4fb45e118403ea9f47, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4763 4491 4498 4491
q2 362 154 165 154
q3 1453 1247 1216 1216
q4 1118 987 924 924
q5 3172 3188 3184 3184
q6 244 131 127 127
q7 992 500 491 491
q8 2193 2204 2171 2171
q9 6671 6624 6628 6624
q10 3183 3245 3263 3245
q11 323 199 217 199
q12 347 216 215 215
q13 5256 3825 3807 3807
q14 243 209 219 209
q15 569 525 526 525
q16 435 387 384 384
q17 1013 604 514 514
q18 7532 7313 7029 7029
q19 1527 1459 1326 1326
q20 735 336 332 332
q21 3090 2643 2661 2643
q22 356 284 285 284
Total cold run time: 45577 ms
Total hot run time: 40094 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4427 4406 4417 4406
q2 270 161 173 161
q3 3516 3508 3524 3508
q4 2377 2366 2364 2364
q5 5717 5737 5730 5730
q6 238 120 123 120
q7 2365 1861 1862 1861
q8 3512 3513 3509 3509
q9 9074 9002 9019 9002
q10 3911 3976 3987 3976
q11 486 397 381 381
q12 771 584 608 584
q13 4306 3575 3513 3513
q14 276 250 246 246
q15 575 518 516 516
q16 497 440 448 440
q17 1857 1874 1834 1834
q18 8735 8289 9210 8289
q19 1754 1736 1745 1736
q20 2254 1935 1948 1935
q21 6447 6149 6125 6125
q22 503 431 417 417
Total cold run time: 63868 ms
Total hot run time: 60653 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #28029:
URL: https://github.com/apache/doris/pull/28029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28029:
URL: https://github.com/apache/doris/pull/28029#discussion_r1415576824
##########
be/src/runtime/load_stream.cpp:
##########
@@ -241,8 +241,11 @@ Status IndexStream::close(const std::vector<PTabletID>& tablets_to_commit,
return Status::OK();
}
+// TODO: Profile is temporary disabled, because:
+// 1. It's not being processed by the upstream for now
+// 2. There are some problems in _profile->to_thrift()
LoadStream::LoadStream(PUniqueId load_id, LoadStreamMgr* load_stream_mgr, bool enable_profile)
- : _load_id(load_id), _enable_profile(enable_profile), _load_stream_mgr(load_stream_mgr) {
+ : _load_id(load_id), _enable_profile(false), _load_stream_mgr(load_stream_mgr) {
Review Comment:
warning: member initializer for '_enable_profile' is redundant [modernize-use-default-member-init]
```suggestion
: _load_id(load_id), , _load_stream_mgr(load_stream_mgr) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28029:
URL: https://github.com/apache/doris/pull/28029#issuecomment-1840943376
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.28 seconds
stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17164485186 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #28029:
URL: https://github.com/apache/doris/pull/28029#issuecomment-1840743072
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28029:
URL: https://github.com/apache/doris/pull/28029#issuecomment-1841909177
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.58 seconds
stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162475931 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #28029:
URL: https://github.com/apache/doris/pull/28029#issuecomment-1840739505
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28029:
URL: https://github.com/apache/doris/pull/28029#issuecomment-1840750851
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) temporary disable load stream profile [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28029:
URL: https://github.com/apache/doris/pull/28029#issuecomment-1840751943
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org