You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "w41ter (via GitHub)" <gi...@apache.org> on 2024/01/11 02:27:21 UTC
[PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
w41ter opened a new pull request, #29833:
URL: https://github.com/apache/doris/pull/29833
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886161025
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.39 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.3 seconds inserted 10000000 Rows, about 469K ops/s
storage size: 17183792597 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886238167
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 66cb6078f0f671baf623af786165e1dcb96cab8d with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.5 seconds inserted 10000000 Rows, about 689K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886232497
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.85 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s
storage size: 17184072411 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886099576
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886234423
TeamCity be ut coverage result:
Function Coverage: 36.70% (8647/23559)
Line Coverage: 28.74% (70440/245135)
Region Coverage: 27.65% (36382/131590)
Branch Coverage: 24.36% (18620/76428)
Coverage Report: http://coverage.selectdb-in.cc/coverage/66cb6078f0f671baf623af786165e1dcb96cab8d_66cb6078f0f671baf623af786165e1dcb96cab8d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #29833:
URL: https://github.com/apache/doris/pull/29833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886236119
<details>
<summary>ClickBench: <b>Total hot run time: 30.61 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66cb6078f0f671baf623af786165e1dcb96cab8d, data reload: false
query1 0.06 0.06 0.05
query2 0.06 0.02 0.03
query3 0.25 0.12 0.10
query4 1.78 0.12 0.11
query5 0.53 0.52 0.51
query6 1.36 0.64 0.63
query7 0.01 0.01 0.02
query8 0.04 0.03 0.03
query9 0.58 0.50 0.49
query10 0.55 0.56 0.55
query11 0.12 0.09 0.08
query12 0.12 0.09 0.09
query13 0.61 0.61 0.61
query14 0.79 0.78 0.79
query15 0.82 0.80 0.80
query16 0.34 0.35 0.35
query17 0.92 0.95 0.99
query18 0.25 0.26 0.23
query19 1.80 1.76 1.77
query20 0.01 0.00 0.01
query21 15.41 0.59 0.56
query22 2.36 2.43 2.43
query23 17.31 0.75 0.84
query24 7.30 1.09 0.30
query25 1.88 0.15 0.15
query26 0.13 0.13 0.14
query27 0.15 0.15 0.14
query28 10.12 0.81 0.78
query29 12.61 3.14 3.22
query30 0.58 0.48 0.48
query31 2.78 0.34 0.37
query32 3.38 0.48 0.49
query33 3.19 3.18 3.25
query34 15.91 4.24 4.14
query35 4.20 4.16 4.18
query36 1.10 1.04 1.05
query37 0.07 0.05 0.05
query38 0.03 0.02 0.02
query39 0.02 0.01 0.01
query40 0.16 0.13 0.14
query41 0.07 0.02 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 109.8 s
Total hot run time: 30.61 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886232025
<details>
<summary>TPC-DS: <b>Total hot run time: 178568 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66cb6078f0f671baf623af786165e1dcb96cab8d, data reload: false
query1 922 345 333 333
query2 6689 1909 1809 1809
query3 6834 219 210 210
query4 26048 22118 22127 22118
query5 5468 537 525 525
query6 274 194 206 194
query7 4627 277 278 277
query8 237 205 208 205
query9 9195 2707 2895 2707
query10 487 253 250 250
query11 16080 15261 15274 15261
query12 135 84 81 81
query13 1635 354 336 336
query14 12128 7423 7366 7366
query15 237 194 206 194
query16 6530 289 267 267
query17 1900 505 500 500
query18 1943 282 285 282
query19 289 152 141 141
query20 89 91 83 83
query21 189 92 96 92
query22 5028 4752 4826 4752
query23 32563 31117 31201 31117
query24 11109 2890 2873 2873
query25 551 348 358 348
query26 922 153 159 153
query27 2795 282 289 282
query28 6347 1853 1873 1853
query29 987 388 415 388
query30 283 148 146 146
query31 969 777 783 777
query32 95 65 58 58
query33 743 276 283 276
query34 897 449 447 447
query35 922 797 774 774
query36 1283 1206 1265 1206
query37 105 70 70 70
query38 3435 3282 3314 3282
query39 1340 1293 1290 1290
query40 207 95 93 93
query41 38 35 36 35
query42 102 89 97 89
query43 570 521 532 521
query44 1097 721 730 721
query45 208 197 193 193
query46 1048 673 668 668
query47 1685 1598 1610 1598
query48 395 308 319 308
query49 1141 326 333 326
query50 743 341 341 341
query51 5407 5244 5173 5173
query52 88 90 88 88
query53 232 153 149 149
query54 1409 571 593 571
query55 97 86 92 86
query56 235 201 211 201
query57 1025 958 961 958
query58 249 225 217 217
query59 2770 2593 2544 2544
query60 263 236 243 236
query61 87 86 85 85
query62 628 402 405 402
query63 174 154 159 154
query64 4619 1745 1757 1745
query65 3360 3294 3266 3266
query66 1044 333 325 325
query67 15655 15526 15215 15215
query68 12342 552 531 531
query69 559 284 288 284
query70 1625 1528 1565 1528
query71 504 232 228 228
query72 5003 2878 2853 2853
query73 2498 323 320 320
query74 6864 6405 6403 6403
query75 5079 2271 2303 2271
query76 6280 930 1130 930
query77 683 294 295 294
query78 9806 8504 8582 8504
query79 1049 515 511 511
query80 621 362 363 362
query81 464 211 211 211
query82 212 92 98 92
query83 171 148 146 146
query84 268 61 57 57
query85 994 312 288 288
query86 416 375 355 355
query87 3549 3433 3401 3401
query88 3265 2257 2240 2240
query89 341 274 268 268
query90 1864 215 212 212
query91 176 134 143 134
query92 65 55 55 55
query93 2152 452 438 438
query94 883 197 192 192
query95 514 437 436 436
query96 644 328 322 322
query97 4244 4177 4200 4177
query98 203 196 184 184
query99 996 710 738 710
Total cold run time: 291220 ms
Total hot run time: 178568 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29833:
URL: https://github.com/apache/doris/pull/29833#discussion_r1448215508
##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -0,0 +1,286 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "cloud/cloud_meta_mgr.h"
+
+#include <brpc/channel.h>
+#include <brpc/controller.h>
+#include <glog/logging.h>
+
+#include <atomic>
+#include <chrono>
+#include <memory>
+#include <mutex>
+#include <random>
+#include <shared_mutex>
+#include <vector>
+
+#include "cloud/config.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "gen_cpp/olap_file.pb.h"
+#include "olap/olap_common.h"
+#include "olap/rowset/rowset_factory.h"
+#include "olap/tablet.h"
+#include "olap/tablet_meta.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/network_util.h"
+#include "util/s3_util.h"
+
+namespace doris::cloud {
+using namespace ErrorCode;
+
+bvar::LatencyRecorder g_get_rowset_latency("doris_CloudMetaMgr", "get_rowset");
+
+class MetaServiceProxy {
+public:
+ static Status get_client(std::shared_ptr<MetaService_Stub>* stub) {
+ SYNC_POINT_RETURN_WITH_VALUE("MetaServiceProxy::get_client", Status::OK(), stub);
+ return get_pooled_client(stub);
+ }
+
+private:
+ static Status get_pooled_client(std::shared_ptr<MetaService_Stub>* stub) {
+ static std::once_flag proxies_flag;
+ static size_t num_proxies = 1;
+ static std::atomic<size_t> index(0);
+ static std::unique_ptr<MetaServiceProxy[]> proxies;
+
+ std::call_once(
+ proxies_flag, +[]() {
+ if (config::meta_service_connection_pooled) {
+ num_proxies = config::meta_service_connection_pool_size;
+ }
+ proxies = std::make_unique<MetaServiceProxy[]>(num_proxies);
+ });
+
+ for (size_t i = 0; i + 1 < num_proxies; ++i) {
+ size_t next_index = index.fetch_add(1, std::memory_order_relaxed) % num_proxies;
+ Status s = proxies[next_index].get(stub);
+ if (s.ok()) return Status::OK();
+ }
+
+ size_t next_index = index.fetch_add(1, std::memory_order_relaxed) % num_proxies;
+ return proxies[next_index].get(stub);
+ }
+
+ static Status init_channel(brpc::Channel* channel) {
+ static std::atomic<size_t> index = 1;
+
+ std::string ip;
+ uint16_t port;
+ Status s = get_meta_service_ip_and_port(&ip, &port);
+ if (!s.ok()) {
+ LOG(WARNING) << "fail to get meta service ip and port: " << s;
+ return s;
+ }
+
+ size_t next_id = index.fetch_add(1, std::memory_order_relaxed);
+ brpc::ChannelOptions options;
+ options.connection_group = fmt::format("ms_{}", next_id);
+ if (channel->Init(ip.c_str(), port, &options) != 0) {
+ return Status::InternalError("fail to init brpc channel, ip: {}, port: {}", ip, port);
+ }
+ return Status::OK();
+ }
+
+ static Status get_meta_service_ip_and_port(std::string* ip, uint16_t* port) {
+ std::string parsed_host;
+ if (!parse_endpoint(config::meta_service_endpoint, &parsed_host, port)) {
+ return Status::InvalidArgument("invalid meta service endpoint: {}",
+ config::meta_service_endpoint);
+ }
+ if (is_valid_ip(parsed_host)) {
+ *ip = std::move(parsed_host);
+ return Status::OK();
+ }
+ return hostname_to_ip(parsed_host, *ip);
+ }
+
+ bool is_idle_timeout(long now) {
Review Comment:
warning: method 'is_idle_timeout' can be made static [readability-convert-member-functions-to-static]
```suggestion
static bool is_idle_timeout(long now) {
```
##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -0,0 +1,286 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "cloud/cloud_meta_mgr.h"
+
+#include <brpc/channel.h>
+#include <brpc/controller.h>
+#include <glog/logging.h>
+
+#include <atomic>
+#include <chrono>
+#include <memory>
+#include <mutex>
+#include <random>
+#include <shared_mutex>
+#include <vector>
+
+#include "cloud/config.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "gen_cpp/olap_file.pb.h"
+#include "olap/olap_common.h"
+#include "olap/rowset/rowset_factory.h"
+#include "olap/tablet.h"
+#include "olap/tablet_meta.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/network_util.h"
+#include "util/s3_util.h"
+
+namespace doris::cloud {
+using namespace ErrorCode;
+
+bvar::LatencyRecorder g_get_rowset_latency("doris_CloudMetaMgr", "get_rowset");
+
+class MetaServiceProxy {
+public:
+ static Status get_client(std::shared_ptr<MetaService_Stub>* stub) {
+ SYNC_POINT_RETURN_WITH_VALUE("MetaServiceProxy::get_client", Status::OK(), stub);
+ return get_pooled_client(stub);
+ }
+
+private:
+ static Status get_pooled_client(std::shared_ptr<MetaService_Stub>* stub) {
+ static std::once_flag proxies_flag;
+ static size_t num_proxies = 1;
+ static std::atomic<size_t> index(0);
+ static std::unique_ptr<MetaServiceProxy[]> proxies;
+
+ std::call_once(
+ proxies_flag, +[]() {
+ if (config::meta_service_connection_pooled) {
+ num_proxies = config::meta_service_connection_pool_size;
+ }
+ proxies = std::make_unique<MetaServiceProxy[]>(num_proxies);
+ });
+
+ for (size_t i = 0; i + 1 < num_proxies; ++i) {
+ size_t next_index = index.fetch_add(1, std::memory_order_relaxed) % num_proxies;
+ Status s = proxies[next_index].get(stub);
+ if (s.ok()) return Status::OK();
+ }
+
+ size_t next_index = index.fetch_add(1, std::memory_order_relaxed) % num_proxies;
+ return proxies[next_index].get(stub);
+ }
+
+ static Status init_channel(brpc::Channel* channel) {
+ static std::atomic<size_t> index = 1;
+
+ std::string ip;
+ uint16_t port;
+ Status s = get_meta_service_ip_and_port(&ip, &port);
+ if (!s.ok()) {
+ LOG(WARNING) << "fail to get meta service ip and port: " << s;
+ return s;
+ }
+
+ size_t next_id = index.fetch_add(1, std::memory_order_relaxed);
+ brpc::ChannelOptions options;
+ options.connection_group = fmt::format("ms_{}", next_id);
+ if (channel->Init(ip.c_str(), port, &options) != 0) {
+ return Status::InternalError("fail to init brpc channel, ip: {}, port: {}", ip, port);
+ }
+ return Status::OK();
+ }
+
+ static Status get_meta_service_ip_and_port(std::string* ip, uint16_t* port) {
+ std::string parsed_host;
+ if (!parse_endpoint(config::meta_service_endpoint, &parsed_host, port)) {
+ return Status::InvalidArgument("invalid meta service endpoint: {}",
+ config::meta_service_endpoint);
+ }
+ if (is_valid_ip(parsed_host)) {
+ *ip = std::move(parsed_host);
+ return Status::OK();
+ }
+ return hostname_to_ip(parsed_host, *ip);
+ }
+
+ bool is_idle_timeout(long now) {
+ auto idle_timeout_ms = config::meta_service_idle_connection_timeout_ms;
+ return idle_timeout_ms > 0 &&
+ _last_access_at_ms.load(std::memory_order_relaxed) + idle_timeout_ms < now;
+ }
+
+ Status get(std::shared_ptr<MetaService_Stub>* stub) {
+ using namespace std::chrono;
+
+ auto now = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ {
+ std::shared_lock lock(_mutex);
+ if (_deadline_ms >= now && !is_idle_timeout(now)) {
+ _last_access_at_ms.store(now, std::memory_order_relaxed);
+ *stub = _stub;
+ return Status::OK();
+ }
+ }
+
+ auto channel = std::make_unique<brpc::Channel>();
+ Status s = init_channel(channel.get());
+ if (UNLIKELY(!s.ok())) {
+ return s;
+ }
+
+ *stub = std::make_shared<MetaService_Stub>(channel.release(),
+ google::protobuf::Service::STUB_OWNS_CHANNEL);
+
+ long deadline = now;
+ if (config::meta_service_connection_age_base_minutes > 0) {
+ std::default_random_engine rng(static_cast<uint32_t>(now));
+ std::uniform_int_distribution<> uni(
+ config::meta_service_connection_age_base_minutes,
+ config::meta_service_connection_age_base_minutes * 2);
+ deadline = now + duration_cast<milliseconds>(minutes(uni(rng))).count();
+ } else {
+ deadline = LONG_MAX;
+ }
+
+ // Last one WIN
+ std::unique_lock lock(_mutex);
+ _last_access_at_ms.store(now, std::memory_order_relaxed);
+ _deadline_ms = deadline;
+ _stub = *stub;
+ return Status::OK();
+ }
+
+ std::shared_mutex _mutex;
+ std::atomic<long> _last_access_at_ms {0};
+ long _deadline_ms {0};
+ std::shared_ptr<MetaService_Stub> _stub;
+};
+
+Status CloudMetaMgr::get_tablet_meta(int64_t tablet_id, TabletMetaSharedPtr* tablet_meta) {
+ VLOG_DEBUG << "send GetTabletRequest, tablet_id: " << tablet_id;
+ TEST_SYNC_POINT_RETURN_WITH_VALUE("CloudMetaMgr::get_tablet_meta", Status::OK(), tablet_id,
+ tablet_meta);
+
+ std::shared_ptr<MetaService_Stub> stub;
+ RETURN_IF_ERROR(MetaServiceProxy::get_client(&stub));
+
+ int tried = 0;
+TRY_AGAIN:
+ brpc::Controller cntl;
+ cntl.set_timeout_ms(config::meta_service_brpc_timeout_ms);
+ GetTabletRequest req;
+ GetTabletResponse resp;
+ req.set_cloud_unique_id(config::cloud_unique_id);
+ req.set_tablet_id(tablet_id);
+ stub->get_tablet(&cntl, &req, &resp, nullptr);
+ int retry_times = config::meta_service_rpc_retry_times;
+ if (cntl.Failed()) {
+ if (tried++ < retry_times) {
+ auto rng = std::default_random_engine(static_cast<uint32_t>(
+ std::chrono::steady_clock::now().time_since_epoch().count()));
+ std::uniform_int_distribution<uint32_t> u(20, 200);
+ std::uniform_int_distribution<uint32_t> u1(500, 1000);
+ uint32_t duration_ms = tried >= 100 ? u(rng) : u1(rng);
+ std::this_thread::sleep_for(std::chrono::milliseconds(duration_ms));
+ LOG_INFO("failed to get tablet meta")
+ .tag("reason", cntl.ErrorText())
+ .tag("tablet_id", tablet_id)
+ .tag("tried", tried)
+ .tag("sleep", duration_ms);
+ goto TRY_AGAIN;
+ }
+ return Status::RpcError("failed to get tablet meta: {}", cntl.ErrorText());
+ }
+ if (resp.status().code() == MetaServiceCode::TABLET_NOT_FOUND) {
+ return Status::NotFound("failed to get tablet meta: {}", resp.status().msg());
+ }
+ if (resp.status().code() != MetaServiceCode::OK) {
+ return Status::InternalError("failed to get tablet meta: {}", resp.status().msg());
+ }
+ *tablet_meta = std::make_shared<TabletMeta>();
+ (*tablet_meta)->init_from_pb(resp.tablet_meta());
+ VLOG_DEBUG << "get tablet meta, tablet_id: " << (*tablet_meta)->tablet_id();
+ return Status::OK();
+}
+
+Status CloudMetaMgr::sync_tablet_rowsets(Tablet* tablet, bool warmup_delta_data) {
+ return Status::NotSupported("CloudMetaMgr::sync_tablet_rowsets is not implemented");
+}
+
+Status CloudMetaMgr::sync_tablet_delete_bitmap(
Review Comment:
warning: method 'sync_tablet_delete_bitmap' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status CloudMetaMgr::sync_tablet_delete_bitmap(
```
##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -0,0 +1,286 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "cloud/cloud_meta_mgr.h"
+
+#include <brpc/channel.h>
+#include <brpc/controller.h>
+#include <glog/logging.h>
+
+#include <atomic>
+#include <chrono>
+#include <memory>
+#include <mutex>
+#include <random>
+#include <shared_mutex>
+#include <vector>
+
+#include "cloud/config.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "gen_cpp/olap_file.pb.h"
+#include "olap/olap_common.h"
+#include "olap/rowset/rowset_factory.h"
+#include "olap/tablet.h"
+#include "olap/tablet_meta.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/network_util.h"
+#include "util/s3_util.h"
+
+namespace doris::cloud {
+using namespace ErrorCode;
+
+bvar::LatencyRecorder g_get_rowset_latency("doris_CloudMetaMgr", "get_rowset");
+
+class MetaServiceProxy {
+public:
+ static Status get_client(std::shared_ptr<MetaService_Stub>* stub) {
+ SYNC_POINT_RETURN_WITH_VALUE("MetaServiceProxy::get_client", Status::OK(), stub);
+ return get_pooled_client(stub);
+ }
+
+private:
+ static Status get_pooled_client(std::shared_ptr<MetaService_Stub>* stub) {
+ static std::once_flag proxies_flag;
+ static size_t num_proxies = 1;
+ static std::atomic<size_t> index(0);
+ static std::unique_ptr<MetaServiceProxy[]> proxies;
+
+ std::call_once(
+ proxies_flag, +[]() {
+ if (config::meta_service_connection_pooled) {
+ num_proxies = config::meta_service_connection_pool_size;
+ }
+ proxies = std::make_unique<MetaServiceProxy[]>(num_proxies);
+ });
+
+ for (size_t i = 0; i + 1 < num_proxies; ++i) {
+ size_t next_index = index.fetch_add(1, std::memory_order_relaxed) % num_proxies;
+ Status s = proxies[next_index].get(stub);
+ if (s.ok()) return Status::OK();
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (s.ok()) { return Status::OK();
}
```
##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -0,0 +1,286 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "cloud/cloud_meta_mgr.h"
+
+#include <brpc/channel.h>
+#include <brpc/controller.h>
+#include <glog/logging.h>
+
+#include <atomic>
+#include <chrono>
+#include <memory>
+#include <mutex>
+#include <random>
+#include <shared_mutex>
+#include <vector>
+
+#include "cloud/config.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "gen_cpp/olap_file.pb.h"
+#include "olap/olap_common.h"
+#include "olap/rowset/rowset_factory.h"
+#include "olap/tablet.h"
+#include "olap/tablet_meta.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/network_util.h"
+#include "util/s3_util.h"
+
+namespace doris::cloud {
+using namespace ErrorCode;
+
+bvar::LatencyRecorder g_get_rowset_latency("doris_CloudMetaMgr", "get_rowset");
+
+class MetaServiceProxy {
+public:
+ static Status get_client(std::shared_ptr<MetaService_Stub>* stub) {
+ SYNC_POINT_RETURN_WITH_VALUE("MetaServiceProxy::get_client", Status::OK(), stub);
+ return get_pooled_client(stub);
+ }
+
+private:
+ static Status get_pooled_client(std::shared_ptr<MetaService_Stub>* stub) {
+ static std::once_flag proxies_flag;
+ static size_t num_proxies = 1;
+ static std::atomic<size_t> index(0);
+ static std::unique_ptr<MetaServiceProxy[]> proxies;
+
+ std::call_once(
+ proxies_flag, +[]() {
+ if (config::meta_service_connection_pooled) {
+ num_proxies = config::meta_service_connection_pool_size;
+ }
+ proxies = std::make_unique<MetaServiceProxy[]>(num_proxies);
+ });
+
+ for (size_t i = 0; i + 1 < num_proxies; ++i) {
+ size_t next_index = index.fetch_add(1, std::memory_order_relaxed) % num_proxies;
+ Status s = proxies[next_index].get(stub);
+ if (s.ok()) return Status::OK();
+ }
+
+ size_t next_index = index.fetch_add(1, std::memory_order_relaxed) % num_proxies;
+ return proxies[next_index].get(stub);
+ }
+
+ static Status init_channel(brpc::Channel* channel) {
+ static std::atomic<size_t> index = 1;
+
+ std::string ip;
+ uint16_t port;
+ Status s = get_meta_service_ip_and_port(&ip, &port);
+ if (!s.ok()) {
+ LOG(WARNING) << "fail to get meta service ip and port: " << s;
+ return s;
+ }
+
+ size_t next_id = index.fetch_add(1, std::memory_order_relaxed);
+ brpc::ChannelOptions options;
+ options.connection_group = fmt::format("ms_{}", next_id);
+ if (channel->Init(ip.c_str(), port, &options) != 0) {
+ return Status::InternalError("fail to init brpc channel, ip: {}, port: {}", ip, port);
+ }
+ return Status::OK();
+ }
+
+ static Status get_meta_service_ip_and_port(std::string* ip, uint16_t* port) {
+ std::string parsed_host;
+ if (!parse_endpoint(config::meta_service_endpoint, &parsed_host, port)) {
+ return Status::InvalidArgument("invalid meta service endpoint: {}",
+ config::meta_service_endpoint);
+ }
+ if (is_valid_ip(parsed_host)) {
+ *ip = std::move(parsed_host);
+ return Status::OK();
+ }
+ return hostname_to_ip(parsed_host, *ip);
+ }
+
+ bool is_idle_timeout(long now) {
+ auto idle_timeout_ms = config::meta_service_idle_connection_timeout_ms;
+ return idle_timeout_ms > 0 &&
+ _last_access_at_ms.load(std::memory_order_relaxed) + idle_timeout_ms < now;
+ }
+
+ Status get(std::shared_ptr<MetaService_Stub>* stub) {
+ using namespace std::chrono;
+
+ auto now = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ {
+ std::shared_lock lock(_mutex);
+ if (_deadline_ms >= now && !is_idle_timeout(now)) {
+ _last_access_at_ms.store(now, std::memory_order_relaxed);
+ *stub = _stub;
+ return Status::OK();
+ }
+ }
+
+ auto channel = std::make_unique<brpc::Channel>();
+ Status s = init_channel(channel.get());
+ if (UNLIKELY(!s.ok())) {
+ return s;
+ }
+
+ *stub = std::make_shared<MetaService_Stub>(channel.release(),
+ google::protobuf::Service::STUB_OWNS_CHANNEL);
+
+ long deadline = now;
+ if (config::meta_service_connection_age_base_minutes > 0) {
+ std::default_random_engine rng(static_cast<uint32_t>(now));
+ std::uniform_int_distribution<> uni(
+ config::meta_service_connection_age_base_minutes,
+ config::meta_service_connection_age_base_minutes * 2);
+ deadline = now + duration_cast<milliseconds>(minutes(uni(rng))).count();
+ } else {
+ deadline = LONG_MAX;
+ }
+
+ // Last one WIN
+ std::unique_lock lock(_mutex);
+ _last_access_at_ms.store(now, std::memory_order_relaxed);
+ _deadline_ms = deadline;
+ _stub = *stub;
+ return Status::OK();
+ }
+
+ std::shared_mutex _mutex;
+ std::atomic<long> _last_access_at_ms {0};
+ long _deadline_ms {0};
+ std::shared_ptr<MetaService_Stub> _stub;
+};
+
+Status CloudMetaMgr::get_tablet_meta(int64_t tablet_id, TabletMetaSharedPtr* tablet_meta) {
+ VLOG_DEBUG << "send GetTabletRequest, tablet_id: " << tablet_id;
+ TEST_SYNC_POINT_RETURN_WITH_VALUE("CloudMetaMgr::get_tablet_meta", Status::OK(), tablet_id,
+ tablet_meta);
+
+ std::shared_ptr<MetaService_Stub> stub;
+ RETURN_IF_ERROR(MetaServiceProxy::get_client(&stub));
+
+ int tried = 0;
+TRY_AGAIN:
+ brpc::Controller cntl;
+ cntl.set_timeout_ms(config::meta_service_brpc_timeout_ms);
+ GetTabletRequest req;
+ GetTabletResponse resp;
+ req.set_cloud_unique_id(config::cloud_unique_id);
+ req.set_tablet_id(tablet_id);
+ stub->get_tablet(&cntl, &req, &resp, nullptr);
+ int retry_times = config::meta_service_rpc_retry_times;
+ if (cntl.Failed()) {
+ if (tried++ < retry_times) {
+ auto rng = std::default_random_engine(static_cast<uint32_t>(
+ std::chrono::steady_clock::now().time_since_epoch().count()));
+ std::uniform_int_distribution<uint32_t> u(20, 200);
+ std::uniform_int_distribution<uint32_t> u1(500, 1000);
+ uint32_t duration_ms = tried >= 100 ? u(rng) : u1(rng);
+ std::this_thread::sleep_for(std::chrono::milliseconds(duration_ms));
+ LOG_INFO("failed to get tablet meta")
+ .tag("reason", cntl.ErrorText())
+ .tag("tablet_id", tablet_id)
+ .tag("tried", tried)
+ .tag("sleep", duration_ms);
+ goto TRY_AGAIN;
+ }
+ return Status::RpcError("failed to get tablet meta: {}", cntl.ErrorText());
+ }
+ if (resp.status().code() == MetaServiceCode::TABLET_NOT_FOUND) {
+ return Status::NotFound("failed to get tablet meta: {}", resp.status().msg());
+ }
+ if (resp.status().code() != MetaServiceCode::OK) {
+ return Status::InternalError("failed to get tablet meta: {}", resp.status().msg());
+ }
+ *tablet_meta = std::make_shared<TabletMeta>();
+ (*tablet_meta)->init_from_pb(resp.tablet_meta());
+ VLOG_DEBUG << "get tablet meta, tablet_id: " << (*tablet_meta)->tablet_id();
+ return Status::OK();
+}
+
+Status CloudMetaMgr::sync_tablet_rowsets(Tablet* tablet, bool warmup_delta_data) {
+ return Status::NotSupported("CloudMetaMgr::sync_tablet_rowsets is not implemented");
+}
+
+Status CloudMetaMgr::sync_tablet_delete_bitmap(
+ Tablet* tablet, int64_t old_max_version,
+ const google::protobuf::RepeatedPtrField<RowsetMetaPB>& rs_metas,
+ const TabletStatsPB& stats, const TabletIndexPB& idx, DeleteBitmap* delete_bitmap) {
+ return Status::NotSupported("CloudMetaMgr::sync_tablet_delete_bitmap is not implemented");
+}
+
+Status CloudMetaMgr::prepare_rowset(const RowsetMeta* rs_meta, bool is_tmp,
+ RowsetMetaSharedPtr* existed_rs_meta) {
+ return Status::NotSupported("CloudMetaMgr::prepare_rowset is not implemented");
+}
+
+Status CloudMetaMgr::commit_rowset(const RowsetMeta* rs_meta, bool is_tmp,
+ RowsetMetaSharedPtr* existed_rs_meta) {
+ return Status::NotSupported("CloudMetaMgr::commit_rowset is not implemented");
+}
+
+Status CloudMetaMgr::update_tmp_rowset(const RowsetMeta& rs_meta) {
+ return Status::NotSupported("CloudMetaMgr::update_tmp_rowset is not implemented");
+}
+
+Status CloudMetaMgr::commit_txn(StreamLoadContext* ctx, bool is_2pc) {
+ return Status::NotSupported("CloudMetaMgr::commit_txn is not implemented");
+}
+
+Status CloudMetaMgr::abort_txn(StreamLoadContext* ctx) {
+ return Status::NotSupported("CloudMetaMgr::abort_txn is not implemented");
+}
+
+Status CloudMetaMgr::precommit_txn(StreamLoadContext* ctx) {
+ return Status::NotSupported("CloudMetaMgr::precommit_txn is not implemented");
+}
+
+Status CloudMetaMgr::get_s3_info(std::vector<std::tuple<std::string, S3Conf>>* s3_infos) {
Review Comment:
warning: method 'get_s3_info' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status CloudMetaMgr::get_s3_info(std::vector<std::tuple<std::string, S3Conf>>* s3_infos) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886223714
<details>
<summary>TPC-H: <b>Total hot run time: 39048 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66cb6078f0f671baf623af786165e1dcb96cab8d, data reload: false
------ Round 1 ----------------------------------
q1 18030 5222 5070 5070
q2 2041 145 135 135
q3 10861 1174 1183 1174
q4 10567 800 844 800
q5 7809 3279 3200 3200
q6 204 125 129 125
q7 897 513 502 502
q8 9279 1985 2007 1985
q9 7430 6481 6481 6481
q10 8258 3057 3105 3057
q11 419 200 200 200
q12 373 195 196 195
q13 18131 3426 3416 3416
q14 246 220 209 209
q15 558 514 515 514
q16 431 385 378 378
q17 958 513 484 484
q18 7584 6826 6706 6706
q19 1599 1407 1357 1357
q20 493 306 311 306
q21 2761 2452 2462 2452
q22 357 309 302 302
Total cold run time: 109286 ms
Total hot run time: 39048 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4979 5035 4913 4913
q2 315 202 203 202
q3 3359 3314 3318 3314
q4 2262 2253 2272 2253
q5 5841 5832 5865 5832
q6 194 120 121 120
q7 2323 1887 1894 1887
q8 3471 3587 3606 3587
q9 8939 8872 8850 8850
q10 3755 3844 3856 3844
q11 552 438 438 438
q12 792 609 601 601
q13 3926 3214 3185 3185
q14 284 276 259 259
q15 580 522 501 501
q16 501 448 457 448
q17 2054 2033 2033 2033
q18 8769 8371 8417 8371
q19 1625 1641 1626 1626
q20 2190 1928 1946 1928
q21 6061 5672 5607 5607
q22 570 468 495 468
Total cold run time: 63342 ms
Total hot run time: 60267 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886149964
<details>
<summary>TPC-H: <b>Total hot run time: 38665 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0a53c991465a50ffe2f8fb4fd820acf6131df22e, data reload: false
------ Round 1 ----------------------------------
q1 17971 5063 5027 5027
q2 2036 148 139 139
q3 10677 1098 1143 1098
q4 10581 753 822 753
q5 7866 3164 3143 3143
q6 203 122 125 122
q7 853 498 488 488
q8 9264 1974 1988 1974
q9 7366 6534 6440 6440
q10 8313 3093 3047 3047
q11 424 226 216 216
q12 359 191 196 191
q13 18124 3447 3400 3400
q14 242 214 204 204
q15 554 521 509 509
q16 438 373 390 373
q17 952 462 487 462
q18 7443 6652 6660 6652
q19 1749 1457 1399 1399
q20 548 301 325 301
q21 2745 2411 2422 2411
q22 360 325 316 316
Total cold run time: 109068 ms
Total hot run time: 38665 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4918 4912 4913 4912
q2 311 195 198 195
q3 3334 3310 3282 3282
q4 2253 2270 2258 2258
q5 5802 5799 5791 5791
q6 194 117 117 117
q7 2310 1834 1829 1829
q8 3441 3527 3555 3527
q9 8840 8858 8859 8858
q10 3744 3788 3820 3788
q11 538 426 421 421
q12 782 607 610 607
q13 4038 3251 3165 3165
q14 290 253 268 253
q15 572 506 511 506
q16 513 474 461 461
q17 2041 2045 2004 2004
q18 8796 8311 8283 8283
q19 1615 1582 1617 1582
q20 2148 1948 1932 1932
q21 6002 5803 5618 5618
q22 585 516 496 496
Total cold run time: 63067 ms
Total hot run time: 59885 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886164217
<details>
<summary>ClickBench: <b>Total hot run time: 30.83 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0a53c991465a50ffe2f8fb4fd820acf6131df22e, data reload: false
query1 0.06 0.06 0.05
query2 0.07 0.03 0.02
query3 0.26 0.12 0.10
query4 1.76 0.12 0.11
query5 0.52 0.52 0.52
query6 1.32 0.64 0.63
query7 0.02 0.01 0.02
query8 0.05 0.03 0.03
query9 0.56 0.52 0.49
query10 0.54 0.54 0.55
query11 0.13 0.09 0.09
query12 0.11 0.10 0.08
query13 0.61 0.60 0.60
query14 0.77 0.82 0.78
query15 0.82 0.80 0.80
query16 0.35 0.36 0.35
query17 0.94 0.93 0.97
query18 0.25 0.26 0.25
query19 1.82 1.75 1.70
query20 0.01 0.01 0.00
query21 15.40 0.58 0.58
query22 2.54 2.44 2.16
query23 17.46 0.85 0.80
query24 5.27 0.64 2.11
query25 1.97 0.15 0.15
query26 0.13 0.14 0.13
query27 0.15 0.14 0.15
query28 9.70 0.81 0.81
query29 12.55 3.23 3.28
query30 0.52 0.49 0.48
query31 2.77 0.36 0.35
query32 3.39 0.48 0.48
query33 3.19 3.23 3.26
query34 15.88 4.16 4.11
query35 4.16 4.17 4.18
query36 1.12 1.04 1.04
query37 0.06 0.04 0.04
query38 0.04 0.03 0.03
query39 0.02 0.01 0.01
query40 0.15 0.12 0.13
query41 0.07 0.02 0.01
query42 0.02 0.02 0.02
query43 0.02 0.02 0.02
Total cold run time: 107.55 s
Total hot run time: 30.83 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886176333
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886166154
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 0a53c991465a50ffe2f8fb4fd820acf6131df22e with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.1 seconds inserted 10000000 Rows, about 709K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886176358
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886167801
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud meta manager skeleton [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29833:
URL: https://github.com/apache/doris/pull/29833#issuecomment-1886158017
<details>
<summary>TPC-DS: <b>Total hot run time: 178234 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0a53c991465a50ffe2f8fb4fd820acf6131df22e, data reload: false
query1 939 345 330 330
query2 6702 1910 1943 1910
query3 6717 208 200 200
query4 26542 22336 22074 22074
query5 5606 492 510 492
query6 280 185 184 184
query7 4618 278 279 278
query8 247 208 208 208
query9 9110 2793 2895 2793
query10 502 243 231 231
query11 16203 15418 15275 15275
query12 142 80 73 73
query13 1625 336 345 336
query14 12113 7272 7244 7244
query15 272 203 192 192
query16 6493 280 267 267
query17 1887 492 482 482
query18 1950 286 266 266
query19 284 144 141 141
query20 89 83 79 79
query21 184 92 90 90
query22 5137 4857 4811 4811
query23 32032 31166 31055 31055
query24 12355 2804 2826 2804
query25 569 340 345 340
query26 1759 154 151 151
query27 2950 285 276 276
query28 7149 1964 1926 1926
query29 2053 392 383 383
query30 291 146 144 144
query31 989 769 798 769
query32 94 65 61 61
query33 740 260 278 260
query34 1067 444 442 442
query35 886 732 737 732
query36 1370 1285 1301 1285
query37 183 69 68 68
query38 3406 3273 3261 3261
query39 1338 1281 1278 1278
query40 297 100 90 90
query41 38 34 32 32
query42 96 95 99 95
query43 548 508 490 490
query44 1098 708 717 708
query45 208 187 181 181
query46 1048 640 640 640
query47 1695 1522 1577 1522
query48 395 320 325 320
query49 1273 327 314 314
query50 812 327 323 323
query51 5355 5239 5272 5239
query52 94 85 88 85
query53 226 157 163 157
query54 1434 578 610 578
query55 98 91 88 88
query56 226 197 197 197
query57 972 943 978 943
query58 229 220 214 214
query59 2793 2613 2638 2613
query60 248 234 218 218
query61 82 84 81 81
query62 668 462 438 438
query63 169 160 152 152
query64 5856 1657 1721 1657
query65 3347 3257 3268 3257
query66 1327 338 323 323
query67 15503 15385 15056 15056
query68 13579 531 513 513
query69 577 277 285 277
query70 2217 1597 1570 1570
query71 505 229 207 207
query72 4848 2839 2832 2832
query73 3540 317 319 317
query74 7022 6457 6456 6456
query75 5478 2304 2254 2254
query76 6426 1050 1108 1050
query77 874 276 268 268
query78 10228 8557 8516 8516
query79 1808 483 491 483
query80 515 347 345 345
query81 416 214 207 207
query82 209 87 96 87
query83 195 137 133 133
query84 269 61 59 59
query85 735 287 282 282
query86 413 361 414 361
query87 3549 3379 3365 3365
query88 2963 2233 2233 2233
query89 337 251 261 251
query90 1863 199 202 199
query91 162 136 131 131
query92 63 52 56 52
query93 1268 427 424 424
query94 937 196 195 195
query95 474 426 415 415
query96 610 331 323 323
query97 4290 4200 4171 4171
query98 202 187 186 186
query99 1080 874 870 870
Total cold run time: 300660 ms
Total hot run time: 178234 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org