You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaijchen (via GitHub)" <gi...@apache.org> on 2023/10/23 12:38:01 UTC
[PR] [don't merge] run move-memtable regression tests [doris]
kaijchen opened a new pull request, #25796:
URL: https://github.com/apache/doris/pull/25796
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #25796:
URL: https://github.com/apache/doris/pull/25796#discussion_r1386714497
##########
be/src/vec/sink/vtablet_sink_v2.cpp:
##########
@@ -210,36 +221,44 @@ Status VOlapTableSinkV2::open(RuntimeState* state) {
SCOPED_CONSUME_MEM_TRACKER(_mem_tracker.get());
signal::set_signal_task_id(_load_id);
- _build_tablet_node_mapping();
- RETURN_IF_ERROR(_open_streams(state->backend_id()));
- _init_row_distribution();
+ RETURN_IF_ERROR(_open_streams());
+ RETURN_IF_ERROR(_init_row_distribution());
return Status::OK();
}
-Status VOlapTableSinkV2::_open_streams(int64_t src_id) {
+Status VOlapTableSinkV2::_init_streams(int64_t src_id) {
for (auto& [dst_id, _] : _tablets_for_node) {
- auto node_info = _nodes_info->find_node(dst_id);
- if (node_info == nullptr) {
- return Status::InternalError("Unknown node {} in tablet location", dst_id);
- }
- std::shared_ptr<Streams> streams;
- streams = ExecEnv::GetInstance()->load_stream_stub_pool()->get_or_create(
- _load_id, src_id, dst_id, _stream_per_node);
- // get tablet schema from each backend only in the 1st stream
- for (auto& stream : *streams | std::ranges::views::take(1)) {
- const std::vector<PTabletID>& tablets_for_schema = _indexes_from_node[node_info->id];
- RETURN_IF_ERROR(stream->open(_state->exec_env()->brpc_internal_client_cache(),
- *node_info, _txn_id, *_schema, tablets_for_schema,
- _total_streams, _state->enable_profile()));
- }
- // for the rest streams, open without getting tablet schema
- for (auto& stream : *streams | std::ranges::views::drop(1)) {
- RETURN_IF_ERROR(stream->open(_state->exec_env()->brpc_internal_client_cache(),
- *node_info, _txn_id, *_schema, {}, _total_streams,
- _state->enable_profile()));
- }
- _streams_for_node[dst_id] = streams;
+ _streams_for_node[dst_id] = ExecEnv::GetInstance()->load_stream_stub_pool()->get_or_create(
+ _load_id, src_id, dst_id, _stream_per_node, _num_local_sink);
+ }
+ return Status::OK();
+}
+
+Status VOlapTableSinkV2::_open_streams() {
+ for (auto& [dst_id, streams] : _streams_for_node) {
+ RETURN_IF_ERROR(_open_streams_to_backend(dst_id, *streams));
+ }
+ return Status::OK();
+}
+
+Status VOlapTableSinkV2::_open_streams_to_backend(int64_t dst_id, ::doris::stream_load::LoadStreams& streams) {
+ auto node_info = _nodes_info->find_node(dst_id);
Review Comment:
warning: 'auto node_info' can be declared as 'const auto *node_info' [readability-qualified-auto]
```suggestion
const auto *node_info = _nodes_info->find_node(dst_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1776362998
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775107716
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775481016
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803951679
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1782670284
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1780318281
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1777132177
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1805823051
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803428630
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.17 seconds
stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162609124 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1801996479
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775101957
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775569963
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.86 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162248613 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1785091117
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1784118779
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.02 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162081343 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1786549599
TeamCity be ut coverage result:
Function Coverage: 37.37% (8470/22663)
Line Coverage: 29.68% (68564/231039)
Region Coverage: 28.26% (35609/126022)
Branch Coverage: 25.12% (18231/72572)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6b44eba0c905eb7c7259a3cc58f1ac949939c6e6_6b44eba0c905eb7c7259a3cc58f1ac949939c6e6/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1801946920
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.64 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162273024 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803043419
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1777553767
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.43 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17161987707 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1781471345
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1805878284
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 11fedbaf578a7f1b686afb814b916308b82dc2cf, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5180 5252 5070 5070
q2 364 191 201 191
q3 2086 2037 2035 2035
q4 1483 1438 1448 1438
q5 4143 4150 4124 4124
q6 255 131 137 131
q7 2071 1602 1624 1602
q8 2754 2756 2763 2756
q9 10357 10294 10159 10159
q10 3540 3567 3551 3551
q11 371 241 253 241
q12 468 295 306 295
q13 4541 4101 4079 4079
q14 323 300 288 288
q15 662 567 600 567
q16 690 633 595 595
q17 1142 1077 1083 1077
q18 7877 7376 7465 7376
q19 1703 1711 1720 1711
q20 614 357 356 356
q21 4933 4594 4598 4594
q22 547 443 440 440
Total cold run time: 56104 ms
Total hot run time: 52676 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5028 5093 4997 4997
q2 367 237 259 237
q3 4056 3991 3908 3908
q4 2747 2733 2754 2733
q5 6539 6537 6561 6537
q6 245 126 131 126
q7 3139 2688 2708 2688
q8 4843 4748 4769 4748
q9 17740 17722 17674 17674
q10 4092 4154 4157 4154
q11 755 655 643 643
q12 989 846 834 834
q13 4283 3890 3876 3876
q14 394 356 356 356
q15 605 556 563 556
q16 797 679 746 679
q17 3843 3867 3930 3867
q18 9344 9124 9298 9124
q19 1802 1799 1795 1795
q20 2398 2073 2071 2071
q21 8811 8709 8704 8704
q22 926 902 928 902
Total cold run time: 83743 ms
Total hot run time: 81209 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1786379370
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.98 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162342340 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803047342
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 0eedfb0f7c7c6414eb01598bbfd1db3560655dea, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5109 4956 5104 4956
q2 361 245 297 245
q3 2096 2034 1970 1970
q4 1444 1424 1401 1401
q5 4068 4091 4082 4082
q6 249 128 130 128
q7 2065 1610 1596 1596
q8 2726 2722 2723 2722
q9 10429 10338 10195 10195
q10 3472 3567 3547 3547
q11 362 248 261 248
q12 459 291 289 289
q13 4488 4087 4096 4087
q14 324 296 285 285
q15 618 552 568 552
q16 696 618 599 599
q17 1119 1073 1074 1073
q18 8025 7504 7413 7413
q19 1707 1676 1696 1676
q20 577 377 354 354
q21 4850 4505 4524 4505
q22 529 440 440 440
Total cold run time: 55773 ms
Total hot run time: 52363 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5052 5011 5038 5011
q2 322 228 246 228
q3 4091 3942 3955 3942
q4 2767 2756 2719 2719
q5 6760 6562 6709 6562
q6 239 124 121 121
q7 3099 2798 2686 2686
q8 4831 4750 4912 4750
q9 17743 17609 17664 17609
q10 4126 4165 4191 4165
q11 797 646 675 646
q12 960 812 799 799
q13 4311 3883 3864 3864
q14 379 348 345 345
q15 626 553 550 550
q16 758 710 723 710
q17 3936 3905 3921 3905
q18 9381 9336 9228 9228
q19 1868 1751 1758 1751
q20 2338 2049 2036 2036
q21 9037 8624 8778 8624
q22 938 851 856 851
Total cold run time: 84359 ms
Total hot run time: 81102 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775462009
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1784143187
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1779725668
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1782717393
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.64 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17161913655 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1776429268
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #25796:
URL: https://github.com/apache/doris/pull/25796#discussion_r1382661621
##########
be/src/runtime/load_stream_writer.cpp:
##########
@@ -121,7 +121,15 @@ Status LoadStreamWriter::close_segment(uint32_t segid) {
return Status::OK();
}
-Status LoadStreamWriter::add_segment(uint32_t segid, SegmentStatistics& stat) {
+Status LoadStreamWriter::add_segment(uint32_t segid, const SegmentStatistics& stat) {
Review Comment:
warning: method 'add_segment' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/load_stream_writer.h:73:
```diff
- Status add_segment(uint32_t segid, const SegmentStatistics& stat);
+ static Status add_segment(uint32_t segid, const SegmentStatistics& stat);
```
##########
be/src/vec/sink/load_stream_stub.cpp:
##########
@@ -83,31 +83,33 @@ void LoadStreamStub::LoadStreamReplyHandler::on_closed(brpc::StreamId id) {
_close_cv.notify_all();
}
-LoadStreamStub::LoadStreamStub(PUniqueId load_id, int64_t src_id)
- : _load_id(load_id),
+LoadStreamStub::LoadStreamStub(PUniqueId load_id, int64_t src_id, int num_use)
+ : _use_cnt(num_use),
+ _load_id(load_id),
_src_id(src_id),
_tablet_schema_for_index(std::make_shared<IndexToTabletSchema>()),
- _enable_unique_mow_for_index(std::make_shared<IndexToEnableMoW>()) {};
+ _enable_unique_mow_for_index(std::make_shared<IndexToEnableMoW>()) {}
LoadStreamStub::LoadStreamStub(LoadStreamStub& stub)
- : _load_id(stub._load_id),
+ : _use_cnt(stub._use_cnt.load()),
+ _load_id(stub._load_id),
_src_id(stub._src_id),
_tablet_schema_for_index(stub._tablet_schema_for_index),
- _enable_unique_mow_for_index(stub._enable_unique_mow_for_index) {};
+ _enable_unique_mow_for_index(stub._enable_unique_mow_for_index) {}
LoadStreamStub::~LoadStreamStub() {
if (_is_init.load() && !_handler.is_closed()) {
brpc::StreamClose(_stream_id);
+ static_cast<void>(close_wait(60000)); // 60s
Review Comment:
warning: 60000 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
```cpp
static_cast<void>(close_wait(60000)); // 60s
^
```
##########
be/src/vec/sink/vtablet_sink_v2.cpp:
##########
@@ -165,35 +175,37 @@
SCOPED_CONSUME_MEM_TRACKER(_mem_tracker.get());
signal::set_signal_task_id(_load_id);
- _build_tablet_node_mapping();
- RETURN_IF_ERROR(_open_streams(state->backend_id()));
-
+ RETURN_IF_ERROR(_open_streams());
return Status::OK();
}
-Status VOlapTableSinkV2::_open_streams(int64_t src_id) {
+Status VOlapTableSinkV2::_init_streams(int64_t src_id) {
for (auto& [dst_id, _] : _tablets_for_node) {
+ _streams_for_node[dst_id] = ExecEnv::GetInstance()->load_stream_stub_pool()->get_or_create(
+ _load_id, src_id, dst_id, _stream_per_node, _num_local_sink);
+ }
+ return Status::OK();
+}
+
+Status VOlapTableSinkV2::_open_streams() {
Review Comment:
warning: method '_open_streams' can be made static [readability-convert-member-functions-to-static]
be/src/vec/sink/vtablet_sink_v2.h:125:
```diff
- Status _open_streams();
+ static Status _open_streams();
```
##########
be/src/vec/sink/vtablet_sink_v2.cpp:
##########
@@ -165,35 +175,37 @@ Status VOlapTableSinkV2::open(RuntimeState* state) {
SCOPED_CONSUME_MEM_TRACKER(_mem_tracker.get());
signal::set_signal_task_id(_load_id);
- _build_tablet_node_mapping();
- RETURN_IF_ERROR(_open_streams(state->backend_id()));
-
+ RETURN_IF_ERROR(_open_streams());
return Status::OK();
}
-Status VOlapTableSinkV2::_open_streams(int64_t src_id) {
+Status VOlapTableSinkV2::_init_streams(int64_t src_id) {
Review Comment:
warning: method '_init_streams' can be made static [readability-convert-member-functions-to-static]
be/src/vec/sink/vtablet_sink_v2.h:123:
```diff
- Status _init_streams(int64_t src_id);
+ static Status _init_streams(int64_t src_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775124070
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1788390669
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1784909265
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.57 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162253024 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1801666146
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit dd7d3f5ae2605aa6b297991f43c19a5667556e3e, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5181 5048 5137 5048
q2 376 219 199 199
q3 2091 2075 2073 2073
q4 1490 1469 1440 1440
q5 4110 4218 4167 4167
q6 253 130 127 127
q7 2064 1587 1613 1587
q8 2800 2775 2777 2775
q9 10567 10307 10346 10307
q10 3538 3602 3612 3602
q11 372 246 261 246
q12 452 303 297 297
q13 4532 4071 4084 4071
q14 328 285 279 279
q15 589 549 560 549
q16 702 624 595 595
q17 1152 1107 1107 1107
q18 7975 7474 7450 7450
q19 1708 1726 1732 1726
q20 614 377 350 350
q21 5001 4604 4614 4604
q22 535 452 428 428
Total cold run time: 56430 ms
Total hot run time: 53027 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4967 4985 4968 4968
q2 348 251 234 234
q3 4015 3985 3985 3985
q4 2806 2748 2779 2748
q5 6571 6566 6545 6545
q6 237 124 120 120
q7 3148 2747 2683 2683
q8 4850 4813 4780 4780
q9 17733 17694 17725 17694
q10 4097 4213 4195 4195
q11 775 648 661 648
q12 1003 833 804 804
q13 4317 3903 3890 3890
q14 398 355 351 351
q15 648 546 544 544
q16 784 709 684 684
q17 3945 3943 3922 3922
q18 9494 9389 9328 9328
q19 1859 1824 1809 1809
q20 2359 2033 2044 2033
q21 9040 8737 8893 8737
q22 946 837 908 837
Total cold run time: 84340 ms
Total hot run time: 81539 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803232106
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.41 seconds
stream load tsv: 572 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162637106 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1802075342
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.96 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162428577 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803871529
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1780310570
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1818185421
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.09 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17097965183 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #25796:
URL: https://github.com/apache/doris/pull/25796#discussion_r1375435220
##########
be/src/vec/sink/delta_writer_v2_pool.cpp:
##########
@@ -36,7 +37,7 @@ DeltaWriterV2* DeltaWriterV2Map::get_or_create(int64_t tablet_id,
return _map.at(tablet_id).get();
}
-Status DeltaWriterV2Map::close() {
+Status DeltaWriterV2Map::close(RuntimeProfile* profile) {
Review Comment:
warning: method 'close' can be made static [readability-convert-member-functions-to-static]
be/src/vec/sink/delta_writer_v2_pool.h:72:
```diff
- Status close(RuntimeProfile* profile);
+ static Status close(RuntimeProfile* profile);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1801658983
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.62 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162537815 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1801928289
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 23c906ee58e8dba12ccd305f5f7b3b4e179852f2, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5296 4999 5159 4999
q2 370 195 198 195
q3 2091 2056 2059 2056
q4 1477 1453 1442 1442
q5 4178 4217 4163 4163
q6 251 136 135 135
q7 2080 1602 1639 1602
q8 2801 2766 2768 2766
q9 10439 10589 10338 10338
q10 3524 3616 3608 3608
q11 380 253 247 247
q12 457 300 300 300
q13 4466 4149 4104 4104
q14 318 296 297 296
q15 598 551 567 551
q16 694 631 603 603
q17 1155 1096 1098 1096
q18 7975 7488 7572 7488
q19 1728 1702 1707 1702
q20 594 358 349 349
q21 4940 4578 4582 4578
q22 532 445 449 445
Total cold run time: 56344 ms
Total hot run time: 53063 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5036 4997 5112 4997
q2 370 233 245 233
q3 4031 3992 3897 3897
q4 2817 2787 2765 2765
q5 6596 6566 6568 6566
q6 242 127 132 127
q7 3151 2731 2749 2731
q8 4840 4803 4803 4803
q9 17858 17834 17791 17791
q10 4141 4238 4205 4205
q11 706 654 655 654
q12 1027 819 811 811
q13 4255 3960 3912 3912
q14 398 358 348 348
q15 601 557 562 557
q16 746 691 728 691
q17 4053 4011 4011 4011
q18 9535 9251 9271 9251
q19 1946 1776 1829 1776
q20 2381 2068 2039 2039
q21 8940 8784 8905 8784
q22 915 860 851 851
Total cold run time: 84585 ms
Total hot run time: 81800 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1784103335
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1784154366
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.63 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17161867871 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1784808262
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.22 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162334950 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775467342
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1789992593
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #25796:
URL: https://github.com/apache/doris/pull/25796#discussion_r1372094706
##########
be/src/olap/memtable_writer.cpp:
##########
@@ -274,41 +278,43 @@ Status MemTableWriter::_do_close_wait() {
return Status::OK();
}
-void MemTableWriter::_update_profile(RuntimeProfile* profile) {
+void MemTableWriter::_init_profile(RuntimeProfile* profile) {
Review Comment:
warning: method '_init_profile' can be made static [readability-convert-member-functions-to-static]
be/src/olap/memtable_writer.h:116:
```diff
- void _init_profile(RuntimeProfile* profile);
+ static void _init_profile(RuntimeProfile* profile);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1805893463
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.7 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162128133 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1777490356
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1780354417
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.66 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162056598 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1783244798
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1784740202
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1793886459
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.56 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162216933 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1801118904
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1801251274
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.43 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162389145 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1801867076
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803121870
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.54 seconds
stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162435759 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803159873
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1801134735
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1793863070
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803404131
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit e705359c2697e4af7475a243e17587d8bf1be21a, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5185 5243 5091 5091
q2 367 190 197 190
q3 2067 2062 2097 2062
q4 1469 1444 1433 1433
q5 4159 4168 4116 4116
q6 251 133 134 133
q7 2081 1622 1627 1622
q8 2745 2726 2734 2726
q9 10343 10257 10186 10186
q10 3505 3586 3584 3584
q11 378 244 244 244
q12 450 311 301 301
q13 4519 4112 4061 4061
q14 327 283 290 283
q15 630 563 559 559
q16 702 621 597 597
q17 1133 1097 1082 1082
q18 7821 7450 7480 7450
q19 1697 1693 1702 1693
q20 578 370 356 356
q21 4949 4563 4570 4563
q22 534 432 444 432
Total cold run time: 55890 ms
Total hot run time: 52764 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5055 4985 4997 4985
q2 337 256 278 256
q3 3979 3949 3943 3943
q4 2768 2760 2782 2760
q5 6497 6487 6522 6487
q6 234 125 126 125
q7 3162 2713 2733 2713
q8 4757 4755 4795 4755
q9 17675 17612 17668 17612
q10 4117 4184 4191 4184
q11 729 692 632 632
q12 977 835 863 835
q13 4316 3912 3869 3869
q14 395 357 358 357
q15 592 536 548 536
q16 806 693 744 693
q17 3958 3953 3951 3951
q18 9503 9185 9278 9185
q19 1784 1777 1767 1767
q20 2367 2078 2041 2041
q21 8889 8757 8799 8757
q22 982 855 859 855
Total cold run time: 83879 ms
Total hot run time: 81298 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1804048469
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 453c5aeb45fb2caa8ffd9274a10ec2db0b7c5d84, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5298 5099 5130 5099
q2 365 189 197 189
q3 2085 2044 2056 2044
q4 1488 1464 1447 1447
q5 4143 4224 4150 4150
q6 256 139 135 135
q7 2122 1606 1621 1606
q8 2804 2781 2794 2781
q9 10629 10379 10261 10261
q10 3510 3567 3586 3567
q11 370 242 244 242
q12 465 305 298 298
q13 4532 4179 4100 4100
q14 336 296 300 296
q15 647 555 577 555
q16 699 628 599 599
q17 1165 1137 1108 1108
q18 7790 7376 7302 7302
q19 1738 1707 1714 1707
q20 612 365 356 356
q21 4957 4598 4598 4598
q22 537 428 436 428
Total cold run time: 56548 ms
Total hot run time: 52868 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4968 4961 5088 4961
q2 355 245 246 245
q3 3986 4013 3962 3962
q4 2815 2746 2779 2746
q5 6546 6534 6515 6515
q6 244 133 130 130
q7 3136 2731 2700 2700
q8 4819 4809 4919 4809
q9 17839 17615 17685 17615
q10 4075 4158 4193 4158
q11 743 650 646 646
q12 1042 820 825 820
q13 4297 3905 3913 3905
q14 388 348 360 348
q15 639 575 568 568
q16 800 708 680 680
q17 3907 3957 3998 3957
q18 9543 9390 9332 9332
q19 1842 1814 1794 1794
q20 2388 2046 2067 2046
q21 8966 8851 8774 8774
q22 965 850 858 850
Total cold run time: 84303 ms
Total hot run time: 81561 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803227283
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit f9577b2005c5fceb03ce725876fb62460e0df9e6, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5220 5104 5192 5104
q2 365 252 224 224
q3 2039 2127 2090 2090
q4 1475 1438 1414 1414
q5 4158 4180 4101 4101
q6 247 131 127 127
q7 2063 1589 1617 1589
q8 2737 2718 2728 2718
q9 10469 10276 10205 10205
q10 3477 3541 3579 3541
q11 378 261 252 252
q12 453 295 297 295
q13 4551 4105 4114 4105
q14 319 287 290 287
q15 635 545 576 545
q16 701 619 592 592
q17 1122 1090 1084 1084
q18 7985 7409 7518 7409
q19 1676 1726 1673 1673
q20 583 360 350 350
q21 4929 4569 4578 4569
q22 527 438 431 431
Total cold run time: 56109 ms
Total hot run time: 52705 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5000 5004 4975 4975
q2 359 231 247 231
q3 4009 3979 3976 3976
q4 2775 2746 2753 2746
q5 6568 6420 6424 6420
q6 238 125 125 125
q7 3111 2732 2654 2654
q8 4743 4738 4703 4703
q9 17750 17526 17585 17526
q10 4110 4204 4183 4183
q11 743 669 677 669
q12 1045 800 797 797
q13 4306 3905 3902 3902
q14 396 350 343 343
q15 605 544 547 544
q16 766 659 692 659
q17 3958 3949 3926 3926
q18 9496 9298 9361 9298
q19 1834 1764 1746 1746
q20 2380 2041 2048 2041
q21 8745 8774 8684 8684
q22 942 843 857 843
Total cold run time: 83879 ms
Total hot run time: 80991 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1804067555
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.33 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162541400 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1804066326
run p0 10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #25796:
URL: https://github.com/apache/doris/pull/25796#discussion_r1379543975
##########
be/src/vec/sink/vtablet_sink_v2.cpp:
##########
@@ -165,35 +173,40 @@ Status VOlapTableSinkV2::open(RuntimeState* state) {
SCOPED_CONSUME_MEM_TRACKER(_mem_tracker.get());
signal::set_signal_task_id(_load_id);
- _build_tablet_node_mapping();
- RETURN_IF_ERROR(_open_streams(state->backend_id()));
-
+ RETURN_IF_ERROR(_open_streams());
return Status::OK();
}
-Status VOlapTableSinkV2::_open_streams(int64_t src_id) {
+void VOlapTableSinkV2::_init_streams(int64_t src_id) {
for (auto& [dst_id, _] : _tablets_for_node) {
+ auto streams = ExecEnv::GetInstance()->load_stream_stub_pool()->get_or_create(
+ _load_id, src_id, dst_id, _stream_per_node);
+ for (auto& stream : *streams) {
+ stream->prepare();
+ }
+ _streams_for_node[dst_id] = streams;
+ }
+}
+
+Status VOlapTableSinkV2::_open_streams() {
Review Comment:
warning: method '_open_streams' can be made static [readability-convert-member-functions-to-static]
be/src/vec/sink/vtablet_sink_v2.h:136:
```diff
- Status _open_streams();
+ static Status _open_streams();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1786518674
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1786564285
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.42 seconds
stream load tsv: 548 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162178320 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1783256644
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775202182
TeamCity be ut coverage result:
Function Coverage: 37.01% (8294/22410)
Line Coverage: 29.16% (66600/228418)
Region Coverage: 27.75% (34595/124686)
Branch Coverage: 24.40% (17570/72010)
Coverage Report: http://coverage.selectdb-in.cc/coverage/64820685f586444211eb18614a25550d9ca68dae_64820685f586444211eb18614a25550d9ca68dae/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1777217134
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.23 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162089128 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1779787198
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.79 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162025826 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1786530434
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1786354044
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1785163037
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.07 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162022633 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803115462
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit d6f22185ee8fbfc0aa42561dada19c0a7d99fd60, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5849 5220 5106 5106
q2 369 203 273 203
q3 2103 2228 2026 2026
q4 1455 1430 1413 1413
q5 4159 4170 4086 4086
q6 250 130 132 130
q7 2029 1625 1611 1611
q8 2717 2702 2713 2702
q9 10524 10316 10177 10177
q10 5272 3542 3557 3542
q11 359 248 253 248
q12 435 295 292 292
q13 4494 4128 4173 4128
q14 321 287 291 287
q15 643 544 564 544
q16 687 614 600 600
q17 1116 1063 1057 1057
q18 7921 7408 7418 7408
q19 1658 1651 1674 1651
q20 569 366 345 345
q21 4901 4495 4516 4495
q22 532 432 433 432
Total cold run time: 58363 ms
Total hot run time: 52483 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5036 5010 4951 4951
q2 335 250 252 250
q3 4022 3978 3968 3968
q4 2786 2766 2775 2766
q5 6472 6393 6392 6392
q6 241 127 127 127
q7 3117 2680 2727 2680
q8 4699 4685 4732 4685
q9 17756 17638 17542 17542
q10 4099 4167 4172 4167
q11 802 637 656 637
q12 1009 818 809 809
q13 4282 3885 3868 3868
q14 385 342 352 342
q15 617 557 536 536
q16 759 699 703 699
q17 3895 4040 3949 3949
q18 9498 9176 9266 9176
q19 1910 1761 1745 1745
q20 2356 2046 2055 2046
q21 8895 8906 8793 8793
q22 966 908 885 885
Total cold run time: 83937 ms
Total hot run time: 81013 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1803366101
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #25796:
URL: https://github.com/apache/doris/pull/25796#discussion_r1389469095
##########
be/src/vec/sink/load_stream_stub.cpp:
##########
@@ -190,19 +203,89 @@ Status LoadStreamStub::add_segment(int64_t partition_id, int64_t index_id, int64
// CLOSE_LOAD
Status LoadStreamStub::close_load(const std::vector<PTabletID>& tablets_to_commit) {
- if (--_num_open > 0) {
+ {
+ std::lock_guard<std::mutex> lock(_tablets_to_commit_mutex);
+ _tablets_to_commit.insert(_tablets_to_commit.end(), tablets_to_commit.begin(),
+ tablets_to_commit.end());
+ }
+ const int num_use = --_use_cnt;
+ LOG(INFO) << "stream " << _stream_id << " close, load_id " << _load_id << " use cnt " << num_use;
+ if (num_use > 0) {
return Status::OK();
}
+ if (num_use < 0) {
+ LOG(WARNING) << "stream " << _stream_id << "already closed, load_id " << _load_id;
+ return Status::InternalError("stream {} already closed", _stream_id);
+ }
+
+ DCHECK(num_use == 0);
+ PStreamHeader header;
+ *header.mutable_load_id() = _load_id;
+ header.set_src_id(_src_id);
+ header.set_opcode(doris::PStreamHeader::CLOSE_LOAD);
+ LOG(INFO) << "send close load on stream " << _stream_id << " load_id " << _load_id;
+ {
+ std::lock_guard<std::mutex> lock(_tablets_to_commit_mutex);
+ for (const auto& tablet : _tablets_to_commit) {
+ *header.add_tablets() = tablet;
+ }
+ }
+ return _encode_and_send(header);
+}
+
+// GET_SCHEMA
+Status LoadStreamStub::get_schema(const std::vector<PTabletID>& tablets) {
PStreamHeader header;
*header.mutable_load_id() = _load_id;
header.set_src_id(_src_id);
header.set_opcode(doris::PStreamHeader::CLOSE_LOAD);
- for (const auto& tablet : tablets_to_commit) {
- *header.add_tablets_to_commit() = tablet;
+ std::ostringstream oss;
+ oss << "fetching tablet schema from stream " << _stream_id << ", load id: " << _load_id
+ << ", tablet id:";
+ for (const auto& tablet : tablets) {
+ *header.add_tablets() = tablet;
+ oss << " " << tablet.tablet_id();
}
+ LOG(INFO) << oss.str();
return _encode_and_send(header);
}
+void LoadStreamStub::add_schema(const std::vector<PTabletSchemaWithIndex>& schemas) {
+ std::lock_guard<bthread::Mutex> lock(_mutex);
+ for (const auto& schema : schemas) {
+ auto tablet_schema = std::make_unique<TabletSchema>();
+ tablet_schema->init_from_pb(schema.tablet_schema());
+ _tablet_schema_for_index->emplace(schema.index_id(), std::move(tablet_schema));
+ _enable_unique_mow_for_index->emplace(schema.index_id(),
+ schema.enable_unique_key_merge_on_write());
+ }
+ _schema_cv.notify_all();
+}
+
+Status LoadStreamStub::wait_for_schema(int64_t partition_id, int64_t index_id, int64_t tablet_id,
+ int64_t timeout_ms) {
+ if (_tablet_schema_for_index->contains(index_id)) {
+ return Status::OK();
+ }
+ PTabletID tablet;
+ tablet.set_partition_id(partition_id);
+ tablet.set_index_id(index_id);
+ tablet.set_tablet_id(tablet_id);
+ RETURN_IF_ERROR(get_schema({tablet}));
+
+ MonotonicStopWatch watch;
+ watch.start();
+ while (!_tablet_schema_for_index->contains(index_id) &&
+ watch.elapsed_time() / 1000 / 1000 < timeout_ms) {
+ static_cast<void>(wait_for_new_schema(100));
Review Comment:
warning: 100 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
```cpp
static_cast<void>(wait_for_new_schema(100));
^
```
##########
be/src/runtime/load_stream.cpp:
##########
@@ -360,6 +360,32 @@ void LoadStream::_report_result(StreamId stream, const Status& st,
}
}
+void LoadStream::_report_schema(StreamId stream, const PStreamHeader& hdr) {
Review Comment:
warning: method '_report_schema' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void LoadStream::_report_schema(StreamId stream, const PStreamHeader& hdr) {
```
##########
be/src/vec/sink/load_stream_stub.cpp:
##########
@@ -215,21 +298,22 @@
buf.append(slice.get_data(), slice.get_size());
}
bool eos = header.opcode() == doris::PStreamHeader::CLOSE_LOAD;
- return _send_with_buffer(buf, eos);
+ bool get_schema = header.opcode() == doris::PStreamHeader::GET_SCHEMA;
+ return _send_with_buffer(buf, eos || get_schema);
}
-Status LoadStreamStub::_send_with_buffer(butil::IOBuf& buf, bool eos) {
+Status LoadStreamStub::_send_with_buffer(butil::IOBuf& buf, bool sync) {
Review Comment:
warning: method '_send_with_buffer' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status LoadStreamStub::_send_with_buffer(butil::IOBuf& buf, bool sync) {
```
##########
be/src/vec/sink/load_stream_stub.h:
##########
@@ -176,6 +185,22 @@ class LoadStreamStub {
return _handler.close_wait(timeout_ms);
}
+ Status wait_for_schema(int64_t partition_id, int64_t index_id, int64_t tablet_id,
+ int64_t timeout_ms = 60000);
Review Comment:
warning: 60000 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
```cpp
int64_t timeout_ms = 60000);
^
```
##########
be/src/vec/sink/load_stream_stub.h:
##########
@@ -176,6 +185,22 @@
return _handler.close_wait(timeout_ms);
}
+ Status wait_for_schema(int64_t partition_id, int64_t index_id, int64_t tablet_id,
+ int64_t timeout_ms = 60000);
+
+ Status wait_for_new_schema(int64_t timeout_ms) {
Review Comment:
warning: method 'wait_for_new_schema' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status wait_for_new_schema(int64_t timeout_ms) {
```
##########
be/src/vec/sink/vtablet_sink_v2.cpp:
##########
@@ -91,7 +92,45 @@ static Status on_partitions_created(void* writer, TCreatePartitionResult* result
return static_cast<VOlapTableSinkV2*>(writer)->on_partitions_created(result);
}
-void VOlapTableSinkV2::_init_row_distribution() {
+Status VOlapTableSinkV2::_incremental_open_streams(
Review Comment:
warning: method '_incremental_open_streams' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status VOlapTableSinkV2::_incremental_open_streams(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1776423990
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775116005
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1776614698
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1777144625
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1781598294
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.03 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162000517 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1818085033
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1818178510
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 2fe00e009defca78c7a52e413d0ea7d5a092ca46, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4954 4656 4670 4656
q2 354 161 158 158
q3 2010 1916 1939 1916
q4 1394 1268 1207 1207
q5 3924 3902 4009 3902
q6 255 129 130 129
q7 1428 907 903 903
q8 2782 2770 2756 2756
q9 9580 9519 9410 9410
q10 3434 3491 3485 3485
q11 380 244 241 241
q12 436 295 292 292
q13 4560 3810 3801 3801
q14 310 285 290 285
q15 584 527 522 522
q16 661 594 575 575
q17 1123 958 915 915
q18 7957 7493 7542 7493
q19 1659 1672 1684 1672
q20 588 299 283 283
q21 4427 3964 3976 3964
q22 481 376 373 373
Total cold run time: 53281 ms
Total hot run time: 48938 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4593 4573 4558 4558
q2 336 229 254 229
q3 4037 4009 3991 3991
q4 2709 2707 2716 2707
q5 9647 9637 9668 9637
q6 247 128 125 125
q7 3058 2498 2493 2493
q8 4434 4444 4430 4430
q9 12912 12896 12784 12784
q10 4062 4142 4136 4136
q11 866 677 665 665
q12 991 818 799 799
q13 4297 3579 3592 3579
q14 375 355 357 355
q15 578 522 526 522
q16 732 684 684 684
q17 3850 3949 3900 3900
q18 9769 9274 9309 9274
q19 1838 1779 1752 1752
q20 2406 2063 2048 2048
q21 8890 8560 8588 8560
q22 927 802 808 802
Total cold run time: 81554 ms
Total hot run time: 78030 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #25796:
URL: https://github.com/apache/doris/pull/25796#discussion_r1386013687
##########
be/src/vec/sink/vtablet_sink_v2.cpp:
##########
@@ -205,35 +215,39 @@ Status VOlapTableSinkV2::open(RuntimeState* state) {
signal::set_signal_task_id(_load_id);
_build_tablet_node_mapping();
- RETURN_IF_ERROR(_open_streams(state->backend_id()));
+ RETURN_IF_ERROR(_open_streams());
_init_row_distribution();
return Status::OK();
}
-Status VOlapTableSinkV2::_open_streams(int64_t src_id) {
+Status VOlapTableSinkV2::_init_streams(int64_t src_id) {
Review Comment:
warning: method '_init_streams' can be made static [readability-convert-member-functions-to-static]
be/src/vec/sink/vtablet_sink_v2.h:127:
```diff
- Status _init_streams(int64_t src_id);
+ static Status _init_streams(int64_t src_id);
```
##########
be/src/vec/sink/vtablet_sink_v2.cpp:
##########
@@ -205,35 +215,39 @@
signal::set_signal_task_id(_load_id);
_build_tablet_node_mapping();
- RETURN_IF_ERROR(_open_streams(state->backend_id()));
+ RETURN_IF_ERROR(_open_streams());
_init_row_distribution();
return Status::OK();
}
-Status VOlapTableSinkV2::_open_streams(int64_t src_id) {
+Status VOlapTableSinkV2::_init_streams(int64_t src_id) {
for (auto& [dst_id, _] : _tablets_for_node) {
+ _streams_for_node[dst_id] = ExecEnv::GetInstance()->load_stream_stub_pool()->get_or_create(
+ _load_id, src_id, dst_id, _stream_per_node, _num_local_sink);
+ }
+ return Status::OK();
+}
+
+Status VOlapTableSinkV2::_open_streams() {
Review Comment:
warning: method '_open_streams' can be made static [readability-convert-member-functions-to-static]
be/src/vec/sink/vtablet_sink_v2.h:129:
```diff
- Status _open_streams();
+ static Status _open_streams();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1801406466
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen closed pull request #25796: [don't merge] run move-memtable regression tests
URL: https://github.com/apache/doris/pull/25796
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775144303
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1775509511
TeamCity be ut coverage result:
Function Coverage: 37.01% (8294/22410)
Line Coverage: 29.16% (66604/228418)
Region Coverage: 27.74% (34589/124686)
Branch Coverage: 24.40% (17573/72010)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cdc0efa5a0716f332297ff18285a8459daa7b4d6_cdc0efa5a0716f332297ff18285a8459daa7b4d6/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1776665664
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.59 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162238057 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1776372353
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1776483722
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.3 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162010704 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1779719875
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1788407711
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1784856894
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1782679059
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1781536826
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [don't merge] run move-memtable regression tests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25796:
URL: https://github.com/apache/doris/pull/25796#issuecomment-1783298444
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.19 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162246669 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org