You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org> on 2024/07/23 07:48:58 UTC
[PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
SWJTU-ZhangLei opened a new pull request, #38243:
URL: https://github.com/apache/doris/pull/38243
Motivation:
* In cloud mode we use `foundationdb` for storing rowset meta, when we load data with many rowsets, `commit_txn` will failed because `foundationdb` currently limits all transactions to be below 10 MB in size, So we spilt `commit_txn` into several sub fdb txn to solve the problem
How:
Now `commit_txn` main flow like this
sub txn 1:
1. update partition `VersionPB` with txn_id
2. submit async task to txn_lazy_committer
txn_lazy_committer:
sub txn 2:
convert tmp rowset meta per batch
sub txn 3:
convert tmp rowset meta per batch
......
sub txn n:
make txn visible and remove txn_id in `VersionPB`
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2250316468
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2303572834
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2304052045
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1722772480
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -618,11 +649,13 @@ void MetaServiceImpl::create_tablets(::google::protobuf::RpcController* controll
msg = fmt::format("failed to get vault id, vault name={}", name);
return;
}
+ DCHECK(request->has_db_id());
Review Comment:
> What happens if the doris FE is the old version?
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1722772720
##########
be/src/cloud/config.h:
##########
@@ -94,4 +94,6 @@ DECLARE_mBool(save_load_error_log_to_s3);
// the theads which sync the datas which loaded in other clusters
DECLARE_mInt32(sync_load_for_tablets_thread);
+DECLARE_mBool(txn_lazy_commit);
Review Comment:
> Add `cloud_` prefix, and add a description too.
done, add a cloud_ prefix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2285947238
<details>
<summary>TPC-H: <b>Total hot run time: 40429 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 814829fad0b51598e72ff1c3c1e78f4f3225389e, data reload: false
------ Round 1 ----------------------------------
q1 17927 4383 4316 4316
q2 2034 184 181 181
q3 11342 1034 1190 1034
q4 10523 790 698 698
q5 7747 2819 2828 2819
q6 222 137 138 137
q7 962 595 606 595
q8 9319 2056 2106 2056
q9 8828 6576 6546 6546
q10 7085 2259 2251 2251
q11 464 249 253 249
q12 392 224 223 223
q13 18735 3010 2985 2985
q14 292 249 235 235
q15 539 475 487 475
q16 643 521 525 521
q17 988 750 656 656
q18 8051 7523 7423 7423
q19 6588 1009 1034 1009
q20 724 419 408 408
q21 5449 4587 4631 4587
q22 1110 1042 1025 1025
Total cold run time: 119964 ms
Total hot run time: 40429 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4492 4280 4336 4280
q2 376 277 270 270
q3 3011 2854 2750 2750
q4 2054 1692 1706 1692
q5 5617 5563 5667 5563
q6 233 140 136 136
q7 2172 1823 1774 1774
q8 3340 3438 3442 3438
q9 8778 8897 8768 8768
q10 3435 3237 3308 3237
q11 625 512 518 512
q12 832 631 637 631
q13 17202 3159 3235 3159
q14 334 293 309 293
q15 534 478 516 478
q16 650 578 573 573
q17 1814 1566 1533 1533
q18 8127 8014 7944 7944
q19 1769 1511 1606 1511
q20 2322 1955 1962 1955
q21 5551 5243 5251 5243
q22 1225 1055 1049 1049
Total cold run time: 74493 ms
Total hot run time: 56789 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1730964155
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ int retry_times = 0;
+ do {
+ LOG(INFO) << "lazy task commit txn_id=" << txn_id_ << " retry_times=" << retry_times;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end =
+ (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id_, txn_id_, txn_kv_, code_, msg_, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code_ != MetaServiceCode::OK) break;
+ }
+ if (code_ != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id_, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code_ = err == TxnErrorCode::TXN_KEY_NOT_FOUND
+ ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id_
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id_
+ << " key=" << hex(ver_key);
+ msg_ = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id_) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id_;
+ msg_ = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ break;
+ }
+ }
+ }
+ make_committed_txn_visible(instance_id_, db_id, txn_id_, txn_kv_, code_, msg_);
+ } while (false);
+ } while (code_ == MetaServiceCode::KV_TXN_CONFLICT &&
+ retry_times++ < config::txn_store_retry_times);
+ {
+ std::unique_lock<std::mutex> lock(mutex_);
+ this->finished_ = true;
Review Comment:
> do we count it as a finish even if `retry_times > config::txn_store_retry_times`? line 413 may be false and make it reach here.
Add more log about code and retry times
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732725106
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ int retry_times = 0;
+ do {
+ LOG(INFO) << "lazy task commit txn_id=" << txn_id_ << " retry_times=" << retry_times;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
Review Comment:
> how come `tmp_rowset_metas.size() == 0` may exist?
multi ms, other ms finish lazy task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2283325550
<details>
<summary>TPC-H: <b>Total hot run time: 39545 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 80ae9bdb333044199b702c9462b10d660e2aa26c, data reload: false
------ Round 1 ----------------------------------
q1 18420 4477 4342 4342
q2 2027 226 209 209
q3 11388 1066 1125 1066
q4 10525 756 678 678
q5 7507 2594 2578 2578
q6 223 137 133 133
q7 960 594 595 594
q8 9220 1924 1964 1924
q9 8804 6595 6603 6595
q10 7000 2187 2178 2178
q11 450 238 249 238
q12 395 229 229 229
q13 18926 3003 2991 2991
q14 286 241 242 241
q15 536 476 493 476
q16 616 530 514 514
q17 989 665 644 644
q18 8133 7642 7347 7347
q19 6246 1070 1070 1070
q20 774 421 419 419
q21 5653 4444 4080 4080
q22 1098 999 1010 999
Total cold run time: 120176 ms
Total hot run time: 39545 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4537 4252 4264 4252
q2 397 274 276 274
q3 2906 2764 2851 2764
q4 2021 1718 1686 1686
q5 5498 5501 5429 5429
q6 225 133 131 131
q7 2198 1781 1774 1774
q8 3272 3421 3444 3421
q9 8707 8869 8800 8800
q10 3495 3193 3283 3193
q11 614 514 510 510
q12 821 690 645 645
q13 17081 3110 3171 3110
q14 310 286 298 286
q15 526 499 481 481
q16 646 562 574 562
q17 1872 1564 1524 1524
q18 8143 7994 7749 7749
q19 4278 1575 1648 1575
q20 2253 1980 1913 1913
q21 5521 5221 5325 5221
q22 1136 1060 1034 1034
Total cold run time: 76457 ms
Total hot run time: 56334 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732486494
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,471 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ // tmp rowset key has been converted
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
Review Comment:
warning: function 'make_committed_txn_visible' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:197:** 82 lines including whitespace and comments (threshold 80)
```cpp
void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
^
```
</details>
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -403,9 +419,9 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
}
}
-void internal_create_tablet(MetaServiceCode& code, std::string& msg,
- const doris::TabletMetaCloudPB& meta, std::shared_ptr<TxnKv> txn_kv,
- const std::string& instance_id,
+void internal_create_tablet(const CreateTabletsRequest* request, MetaServiceCode& code,
Review Comment:
warning: function 'internal_create_tablet' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void internal_create_tablet(const CreateTabletsRequest* request, MetaServiceCode& code,
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/meta_service.cpp:421:** 130 lines including whitespace and comments (threshold 80)
```cpp
void internal_create_tablet(const CreateTabletsRequest* request, MetaServiceCode& code,
^
```
</details>
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,471 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
Review Comment:
warning: function 'convert_tmp_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void convert_tmp_rowsets(
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:40:** 151 lines including whitespace and comments (threshold 80)
```cpp
void convert_tmp_rowsets(
^
```
</details>
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,471 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ // tmp rowset key has been converted
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(recycle_key, recycle_val);
+ LOG(INFO) << "put recycle_key=" << hex(recycle_key) << " txn_id=" << txn_id;
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:293:** 137 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1713020176
##########
cloud/src/common/config.h:
##########
@@ -201,4 +201,8 @@ CONF_Validator(s3_client_http_scheme, [](const std::string& config) -> bool {
// Max retry times for object storage request
CONF_mInt64(max_s3_client_retry, "10");
+CONF_Bool(enable_txn_lazy_commit, "true");
+CONF_Int32(txn_lazy_commit_rowsets_thresold, "1024");
+CONF_Int32(txn_lazy_commit_worker_num, "8");
Review Comment:
naming `txn_lazy_commit_num_threads`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2259595020
<details>
<summary>ClickBench: <b>Total hot run time: 30.16 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8ed44221f04cb0a14f7132c207fbd3cf93e572f1, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.04 0.04
query3 0.22 0.05 0.05
query4 1.67 0.07 0.07
query5 0.50 0.49 0.48
query6 1.15 0.71 0.71
query7 0.02 0.01 0.01
query8 0.06 0.04 0.05
query9 0.56 0.52 0.52
query10 0.57 0.56 0.56
query11 0.16 0.13 0.12
query12 0.15 0.12 0.14
query13 0.61 0.60 0.60
query14 0.78 0.78 0.82
query15 0.90 0.86 0.86
query16 0.37 0.36 0.36
query17 1.00 1.00 0.97
query18 0.23 0.21 0.22
query19 1.83 1.74 1.77
query20 0.01 0.00 0.01
query21 15.40 0.77 0.66
query22 4.67 6.67 1.31
query23 17.94 1.31 1.31
query24 2.29 0.22 0.22
query25 0.19 0.09 0.08
query26 0.32 0.22 0.21
query27 0.45 0.23 0.23
query28 13.16 0.99 0.97
query29 12.62 3.35 3.32
query30 0.26 0.06 0.06
query31 2.86 0.40 0.41
query32 3.25 0.50 0.47
query33 2.96 2.96 3.03
query34 15.47 4.31 4.27
query35 4.30 4.31 4.34
query36 0.68 0.49 0.49
query37 0.19 0.16 0.17
query38 0.16 0.16 0.16
query39 0.05 0.04 0.04
query40 0.17 0.13 0.13
query41 0.10 0.04 0.05
query42 0.06 0.06 0.05
query43 0.05 0.05 0.04
Total cold run time: 108.5 s
Total hot run time: 30.16 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2260550318
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1708565487
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -917,320 +1461,418 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
- msg = ss.str();
- return;
- }
- version = version_pb.version();
- } else {
- version = 1;
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
}
- new_versions[version_keys[i]] = version + 1;
+ txn->put(recycle_key, recycle_val);
}
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ err = txn->commit();
Review Comment:
> No need to commit if this TXN is in TXN_STATUS_VISIBLE
done
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ break;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t, std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end = (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id_, txn_id_, txn_kv_, code_, msg_, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code_ != MetaServiceCode::OK) break;
+ }
+ if (code_ != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id_, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code_ = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id_
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id_ << " key=" << hex(ver_key);
+ msg_ = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id_) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id_;
+ msg_ = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+ }
+
+ err = txn->commit();
Review Comment:
> No need to commit if `version_pb.has_txn_id() && version_pb.txn_id() == txn_id_` is false.
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2283493319
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1724473528
##########
cloud/test/txn_lazy_commit_test.cpp:
##########
@@ -0,0 +1,214 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <brpc/controller.h>
+#include <bvar/window.h>
+#include <fmt/core.h>
+#include <gen_cpp/cloud.pb.h>
+#include <gen_cpp/olap_file.pb.h>
+#include <google/protobuf/repeated_field.h>
+#include <gtest/gtest.h>
+
+#include <atomic>
+#include <condition_variable>
+#include <cstdint>
+#include <memory>
+#include <random>
+#include <string>
+#include <thread>
+
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/util.h"
+#include "cpp/sync_point.h"
+#include "meta-service/keys.h"
+#include "meta-service/mem_txn_kv.h"
+#include "meta-service/meta_service.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/txn_kv_error.h"
+#include "mock_resource_manager.h"
+#include "rate-limiter/rate_limiter.h"
+#include "resource-manager/resource_manager.h"
+
+int main(int argc, char** argv) {
+ const std::string conf_file = "doris_cloud.conf";
+ if (!doris::cloud::config::init(conf_file.c_str(), true)) {
+ std::cerr << "failed to init config file, conf=" << conf_file << std::endl;
+ return -1;
+ }
+
+ config::enable_txn_lazy_commit = true;
+ config::txn_lazy_commit_rowsets_thresold = 1;
+ config::txn_lazy_max_rowsets_per_batch = 1;
+ config::txn_lazy_commit_num_threads = 2;
+
+ if (!doris::cloud::init_glog("txn_lazy_commit_test")) {
+ std::cerr << "failed to init glog" << std::endl;
+ return -1;
+ }
+ ::testing::InitGoogleTest(&argc, argv);
+ return RUN_ALL_TESTS();
+}
+namespace doris::cloud {
+
+std::unique_ptr<MetaServiceProxy> get_meta_service(std::shared_ptr<TxnKv> txn_kv,
+ bool mock_resource_mgr) {
+ std::unique_ptr<Transaction> txn;
+ EXPECT_EQ(txn_kv->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->remove("\x00", "\xfe"); // This is dangerous if the fdb is not correctly set
+ EXPECT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
+
+ auto rs = mock_resource_mgr ? std::make_shared<MockResourceManager>(txn_kv)
+ : std::make_shared<ResourceManager>(txn_kv);
+ auto rl = std::make_shared<RateLimiter>();
+ auto meta_service = std::make_unique<MetaServiceImpl>(txn_kv, rs, rl);
+ return std::make_unique<MetaServiceProxy>(std::move(meta_service));
+}
+
+static std::string next_rowset_id() {
+ static int cnt = 0;
+ return std::to_string(++cnt);
+}
+
+static void add_tablet(CreateTabletsRequest& req, int64_t table_id, int64_t index_id,
+ int64_t partition_id, int64_t tablet_id) {
+ auto tablet = req.add_tablet_metas();
+ tablet->set_table_id(table_id);
+ tablet->set_index_id(index_id);
+ tablet->set_partition_id(partition_id);
+ tablet->set_tablet_id(tablet_id);
+ auto schema = tablet->mutable_schema();
+ schema->set_schema_version(0);
+ auto first_rowset = tablet->add_rs_metas();
+ first_rowset->set_rowset_id(0); // required
+ first_rowset->set_rowset_id_v2(next_rowset_id());
+ first_rowset->set_start_version(0);
+ first_rowset->set_end_version(1);
+ first_rowset->mutable_tablet_schema()->CopyFrom(*schema);
+}
+
+static void create_tablet(MetaServiceProxy* meta_service, int64_t table_id, int64_t index_id,
+ int64_t partition_id, int64_t tablet_id) {
+ brpc::Controller cntl;
+ CreateTabletsRequest req;
+ CreateTabletsResponse res;
+ add_tablet(req, table_id, index_id, partition_id, tablet_id);
+ meta_service->create_tablets(&cntl, &req, &res, nullptr);
+ ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << tablet_id;
+}
+
+static doris::RowsetMetaCloudPB create_rowset(int64_t txn_id, int64_t tablet_id,
+ int partition_id = 10, int64_t version = -1,
+ int num_rows = 100) {
+ doris::RowsetMetaCloudPB rowset;
+ rowset.set_rowset_id(0); // required
+ rowset.set_rowset_id_v2(next_rowset_id());
+ rowset.set_tablet_id(tablet_id);
+ rowset.set_partition_id(partition_id);
+ rowset.set_txn_id(txn_id);
+ if (version > 0) {
+ rowset.set_start_version(version);
+ rowset.set_end_version(version);
+ }
+ rowset.set_num_segments(1);
+ rowset.set_num_rows(num_rows);
+ rowset.set_data_disk_size(num_rows * 100);
+ rowset.mutable_tablet_schema()->set_schema_version(0);
+ rowset.set_txn_expiration(::time(nullptr)); // Required by DCHECK
+ return rowset;
+}
+
+static void commit_rowset(MetaServiceProxy* meta_service, const doris::RowsetMetaCloudPB& rowset,
+ CreateRowsetResponse& res) {
+ brpc::Controller cntl;
+ auto arena = res.GetArena();
+ auto req = google::protobuf::Arena::CreateMessage<CreateRowsetRequest>(arena);
+ req->mutable_rowset_meta()->CopyFrom(rowset);
+ meta_service->commit_rowset(&cntl, req, &res, nullptr);
+ if (!arena) delete req;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!arena) { delete req;
}
```
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -403,9 +434,9 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
}
}
-void internal_create_tablet(MetaServiceCode& code, std::string& msg,
- const doris::TabletMetaCloudPB& meta, std::shared_ptr<TxnKv> txn_kv,
- const std::string& instance_id,
+void internal_create_tablet(const CreateTabletsRequest* request, MetaServiceCode& code,
Review Comment:
warning: function 'internal_create_tablet' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void internal_create_tablet(const CreateTabletsRequest* request, MetaServiceCode& code,
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/meta_service.cpp:436:** 130 lines including whitespace and comments (threshold 80)
```cpp
void internal_create_tablet(const CreateTabletsRequest* request, MetaServiceCode& code,
^
```
</details>
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:292:** 126 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2298087733
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2296748712
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2298398447
<details>
<summary>TPC-DS: <b>Total hot run time: 196613 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1d63b327147196a6029b3421305f1eed1a00dc18, data reload: false
query1 1302 889 891 889
query2 6558 2012 1940 1940
query3 10662 3872 3599 3599
query4 56834 23811 23250 23250
query5 5779 730 727 727
query6 556 215 210 210
query7 6406 334 337 334
query8 558 438 440 438
query9 9321 2531 2518 2518
query10 588 349 340 340
query11 17888 14983 15301 14983
query12 200 147 146 146
query13 1705 488 448 448
query14 11903 6914 7511 6914
query15 260 201 204 201
query16 7505 554 529 529
query17 1182 606 623 606
query18 2028 352 339 339
query19 308 174 172 172
query20 156 139 136 136
query21 259 148 143 143
query22 5026 4640 4638 4638
query23 34688 33877 33919 33877
query24 5698 3005 3003 3003
query25 585 432 427 427
query26 721 183 188 183
query27 1867 306 307 306
query28 4045 2187 2179 2179
query29 718 457 451 451
query30 233 184 204 184
query31 1037 851 792 792
query32 97 75 80 75
query33 545 355 340 340
query34 903 502 493 493
query35 858 770 781 770
query36 1096 997 968 968
query37 156 104 103 103
query38 3898 3968 3902 3902
query39 1537 1500 1455 1455
query40 237 157 151 151
query41 139 138 138 138
query42 133 114 116 114
query43 528 505 504 504
query44 1114 781 789 781
query45 231 197 197 197
query46 1112 795 781 781
query47 1912 1858 1872 1858
query48 406 332 333 332
query49 937 598 577 577
query50 863 466 475 466
query51 7192 7230 7107 7107
query52 120 109 105 105
query53 296 228 229 228
query54 617 503 497 497
query55 89 87 90 87
query56 330 313 312 312
query57 1207 1141 1124 1124
query58 295 311 306 306
query59 2885 2791 2861 2791
query60 345 330 323 323
query61 149 146 147 146
query62 791 689 672 672
query63 259 234 228 228
query64 3388 1882 1862 1862
query65 3272 3184 3201 3184
query66 1032 696 667 667
query67 15332 15027 15052 15027
query68 5210 586 580 580
query69 643 310 315 310
query70 1217 1211 1161 1161
query71 544 311 313 311
query72 7683 2379 2084 2084
query73 820 356 356 356
query74 9368 8968 8854 8854
query75 4506 2781 2756 2756
query76 3458 1053 1100 1053
query77 839 440 425 425
query78 10020 9154 9219 9154
query79 7492 556 556 556
query80 1423 600 618 600
query81 621 268 265 265
query82 1222 156 157 156
query83 402 210 207 207
query84 302 97 97 97
query85 1250 358 360 358
query86 455 317 325 317
query87 4393 4288 4138 4138
query88 5536 2465 2476 2465
query89 445 328 325 325
query90 2127 230 237 230
query91 150 127 126 126
query92 86 77 77 77
query93 6062 551 556 551
query94 971 321 326 321
query95 383 300 298 298
query96 624 287 287 287
query97 3211 3144 3111 3111
query98 239 226 223 223
query99 1571 1288 1318 1288
Total cold run time: 334150 ms
Total hot run time: 196613 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732235705
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
- // Save table versions
- for (auto& i : table_id_tablet_ids) {
- std::string ver_key = table_version_key({instance_id, db_id, i.first});
- txn->atomic_add(ver_key, 1);
- LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key) << " txn_id=" << txn_id;
- }
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << "txn_id=" << txn_id << " table_id=" << table_id
+ << " partition_id=" << partition_id << " version=" << i.second;
- LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
- // Update txn_info
- txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- auto now_time = system_clock::now();
- uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
- if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
- LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
- << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
- return;
- }
- txn_info.set_commit_time(commit_time);
- txn_info.set_finish_time(commit_time);
- if (request->has_commit_attachment()) {
- txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
- }
- LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
- info_val.clear();
- if (!txn_info.SerializeToString(&info_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(info_key, info_val);
- LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+ for (size_t i = 0; i < lock_keys.size(); i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- // Update stats of affected tablet
- for (auto& [tablet_id, stats] : tablet_stats) {
- DCHECK(tablet_ids.count(tablet_id));
- auto& tablet_idx = tablet_ids[tablet_id];
- StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
- tablet_idx.partition_id(), tablet_id};
- update_tablet_stats(info, stats, txn, code, msg);
- if (code != MetaServiceCode::OK) return;
- }
- // Remove tmp rowset meta
- for (auto& [k, _] : tmp_rowsets_meta) {
- txn->remove(k);
- LOG(INFO) << "xxx remove tmp_rowset_key=" << hex(k) << " txn_id=" << txn_id;
- }
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
- const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
- LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
- txn->remove(running_key);
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
- std::string recycle_val;
- std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
- RecycleTxnPB recycle_pb;
- recycle_pb.set_creation_time(commit_time);
- recycle_pb.set_label(txn_info.label());
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
+ << " txn_id=" << txn_id;
+ }
- if (!recycle_pb.SerializeToString(&recycle_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(recycle_key, recycle_val);
+ LOG(INFO) << "commit_txn_eventually put_size=" << txn->put_bytes()
+ << " del_size=" << txn->delete_bytes() << " num_put_keys=" << txn->num_put_keys()
+ << " num_del_keys=" << txn->num_del_keys()
+ << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
- if (txn_info.load_job_source_type() ==
- LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
- put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
- }
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
- LOG(INFO) << "xxx commit_txn put recycle_key key=" << hex(recycle_key) << " txn_id=" << txn_id;
- LOG(INFO) << "commit_txn put_size=" << txn->put_bytes() << " del_size=" << txn->delete_bytes()
- << " num_put_keys=" << txn->num_put_keys() << " num_del_keys=" << txn->num_del_keys()
- << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer->submit(instance_id, txn_id);
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ if (ret.first != MetaServiceCode::OK) {
+ LOG(WARNING) << "lazy commit txn failed txn_id=" << txn_id << " code=" << ret.first
+ << "msg=" << ret.second;
+ }
- // Finally we are done...
- err = txn->commit();
- if (err != TxnErrorCode::TXN_OK) {
- if ((err == TxnErrorCode::TXN_VALUE_TOO_LARGE ||
- err == TxnErrorCode::TXN_BYTES_TOO_LARGE) &&
- !tmp_rowsets_meta.empty()) {
- size_t max_size = 0, max_idx = 0, max_num_segments = 0,
- min_num_segments = std::numeric_limits<size_t>::max(), avg_num_segments = 0;
- for (size_t i = 0; i < tmp_rowsets_meta.size(); i++) {
- auto& [k, v] = tmp_rowsets_meta[i];
- if (v.ByteSizeLong() > max_size) {
- max_size = v.ByteSizeLong();
- max_idx = i;
- }
- if (v.num_segments() > max_num_segments) {
- max_num_segments = v.num_segments();
- }
- if (v.num_segments() < min_num_segments) {
- min_num_segments = v.num_segments();
- }
- avg_num_segments += v.num_segments();
- }
- if (!tmp_rowsets_meta.empty()) {
- avg_num_segments /= tmp_rowsets_meta.size();
- }
- LOG(WARNING) << "failed to commit kv txn"
- << ", err=" << err << ", txn_id=" << txn_id
- << ", total_rowsets=" << tmp_rowsets_meta.size()
- << ", avg_num_segments=" << avg_num_segments
- << ", min_num_segments=" << min_num_segments
- << ", max_num_segments=" << max_num_segments
- << ", largest_rowset_size=" << max_size
- << ", largest_rowset_key=" << hex(tmp_rowsets_meta[max_idx].first)
- << ", largest_rowset_value="
- << tmp_rowsets_meta[max_idx].second.ShortDebugString();
+ std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ // Accumulate affected rows
+ auto& stats = tablet_stats[i.tablet_id()];
+ stats.data_size += i.data_disk_size();
+ stats.num_rows += i.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += i.num_segments();
}
- code = cast_as<ErrCategory::COMMIT>(err);
- ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- return;
- }
- // calculate table stats from tablets stats
- std::map<int64_t /*table_id*/, TableStats> table_stats;
- std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
- request->base_tablet_ids().end());
- calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
- for (const auto& pair : table_stats) {
- TableStatsPB* stats_pb = response->add_table_stats();
- auto table_id = pair.first;
- auto stats = pair.second;
- get_pb_from_tablestats(stats, stats_pb);
- stats_pb->set_table_id(table_id);
- VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
- << " table_id=" << table_id
- << " updated_row_count=" << stats_pb->updated_row_count();
- }
+ // calculate table stats from tablets stats
+ std::map<int64_t /*table_id*/, TableStats> table_stats;
+ std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
+ request->base_tablet_ids().end());
+ calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
+ for (const auto& pair : table_stats) {
+ TableStatsPB* stats_pb = response->add_table_stats();
+ auto table_id = pair.first;
+ auto stats = pair.second;
+ get_pb_from_tablestats(stats, stats_pb);
+ stats_pb->set_table_id(table_id);
+ VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
+ << " table_id=" << table_id
+ << " updated_row_count=" << stats_pb->updated_row_count();
+ }
- response->mutable_txn_info()->CopyFrom(txn_info);
-} // end commit_txn
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
Review Comment:
> add comment to describe why it is OK to set VISIBLE
Add a comment, this is used for fe callback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312796154
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1730632559
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1843,11 +1997,12 @@ int InstanceRecycler::scan_and_recycle(
return ret;
}
-int InstanceRecycler::abort_timeout_txn() {
- const std::string task_name = "abort_timeout_txn";
+int InstanceRecycler::advance_pending_txn() {
Review Comment:
keep the old name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2314623553
<details>
<summary>TPC-H: <b>Total hot run time: 38018 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0f6412f7fa9a90fc50e6af06a04d4e9b42655491, data reload: false
------ Round 1 ----------------------------------
q1 18641 6512 4348 4348
q2 2027 187 175 175
q3 11731 927 1167 927
q4 10429 762 784 762
q5 7734 2851 2828 2828
q6 227 139 135 135
q7 978 612 597 597
q8 9340 2102 2094 2094
q9 7220 6535 6556 6535
q10 6999 2273 2219 2219
q11 466 249 255 249
q12 401 224 228 224
q13 18090 3009 3042 3009
q14 284 241 242 241
q15 518 481 490 481
q16 575 510 521 510
q17 991 676 713 676
q18 7504 6750 6894 6750
q19 1404 1097 979 979
q20 690 333 333 333
q21 3979 3097 2923 2923
q22 1134 1023 1040 1023
Total cold run time: 111362 ms
Total hot run time: 38018 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4371 4339 4267 4267
q2 373 274 269 269
q3 2889 2698 2647 2647
q4 1904 1722 1698 1698
q5 5698 5689 5758 5689
q6 223 131 131 131
q7 2224 1826 1851 1826
q8 3287 3449 3521 3449
q9 8854 8835 8861 8835
q10 3619 3413 3347 3347
q11 592 521 496 496
q12 847 661 670 661
q13 14093 3119 3253 3119
q14 339 289 289 289
q15 540 499 477 477
q16 625 590 576 576
q17 1871 1533 1486 1486
q18 8244 7927 7923 7923
q19 1758 1675 1565 1565
q20 2186 1876 1935 1876
q21 5852 5440 5415 5415
q22 1151 1091 1077 1077
Total cold run time: 71540 ms
Total hot run time: 57118 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2314663277
<details>
<summary>ClickBench: <b>Total hot run time: 32.53 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0f6412f7fa9a90fc50e6af06a04d4e9b42655491, data reload: false
query1 0.05 0.04 0.04
query2 0.08 0.03 0.04
query3 0.23 0.05 0.04
query4 1.68 0.07 0.07
query5 0.49 0.48 0.50
query6 1.13 0.72 0.73
query7 0.02 0.01 0.01
query8 0.06 0.04 0.04
query9 0.55 0.49 0.49
query10 0.55 0.54 0.54
query11 0.15 0.12 0.12
query12 0.15 0.12 0.11
query13 0.62 0.60 0.59
query14 2.02 2.07 2.10
query15 0.84 0.83 0.83
query16 0.38 0.36 0.37
query17 0.98 0.98 1.02
query18 0.20 0.20 0.20
query19 1.84 1.81 1.77
query20 0.01 0.01 0.01
query21 15.40 0.67 0.66
query22 4.64 6.18 2.61
query23 18.29 1.32 1.25
query24 2.10 0.22 0.23
query25 0.15 0.08 0.08
query26 0.28 0.18 0.17
query27 0.08 0.08 0.07
query28 13.25 1.02 0.99
query29 12.59 3.35 3.30
query30 0.24 0.06 0.05
query31 2.86 0.39 0.40
query32 3.27 0.48 0.47
query33 2.98 2.93 3.03
query34 17.29 4.35 4.41
query35 4.47 4.49 4.41
query36 0.66 0.48 0.47
query37 0.19 0.17 0.15
query38 0.16 0.15 0.15
query39 0.05 0.04 0.04
query40 0.15 0.12 0.12
query41 0.10 0.06 0.06
query42 0.06 0.05 0.06
query43 0.05 0.04 0.04
Total cold run time: 111.34 s
Total hot run time: 32.53 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732711244
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
Review Comment:
> add comment what does it do and why
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732716284
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,478 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
Review Comment:
warning: function 'convert_tmp_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void convert_tmp_rowsets(
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:40:** 158 lines including whitespace and comments (threshold 80)
```cpp
void convert_tmp_rowsets(
^
```
</details>
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,478 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+ DCHECK(version_pb.txn_ids_size() > 0);
+ DCHECK_EQ(version_pb.txn_ids(0), txn_id);
+ int64_t version = 0;
+ if (version_pb.has_version()) {
+ version = version_pb.version() + 1;
+ } else {
+ version = 2;
+ }
+
+ std::string rowset_key = meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ // tmp rowset key has been converted
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version);
+ tmp_rowset_pb.set_end_version(version);
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
Review Comment:
warning: function 'make_committed_txn_visible' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:204:** 82 lines including whitespace and comments (threshold 80)
```cpp
void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
^
```
</details>
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,478 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+ DCHECK(version_pb.txn_ids_size() > 0);
+ DCHECK_EQ(version_pb.txn_ids(0), txn_id);
+ int64_t version = 0;
+ if (version_pb.has_version()) {
+ version = version_pb.version() + 1;
+ } else {
+ version = 2;
+ }
+
+ std::string rowset_key = meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ // tmp rowset key has been converted
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version);
+ tmp_rowset_pb.set_end_version(version);
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(recycle_key, recycle_val);
+ LOG(INFO) << "put recycle_key=" << hex(recycle_key) << " txn_id=" << txn_id;
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:300:** 137 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732712524
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2250637297
<details>
<summary>ClickBench: <b>Total hot run time: 30.73 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9511fed37daee25d5ec0c6d5a1033b69e5eddf44, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.03 0.04
query3 0.22 0.05 0.06
query4 1.67 0.10 0.10
query5 0.51 0.49 0.52
query6 1.15 0.74 0.73
query7 0.02 0.01 0.02
query8 0.05 0.04 0.05
query9 0.55 0.50 0.49
query10 0.55 0.54 0.55
query11 0.15 0.10 0.11
query12 0.15 0.12 0.12
query13 0.59 0.58 0.58
query14 0.75 0.78 0.77
query15 0.86 0.81 0.81
query16 0.36 0.37 0.36
query17 1.03 1.03 1.00
query18 0.23 0.22 0.22
query19 1.82 1.81 1.79
query20 0.01 0.00 0.01
query21 15.39 0.78 0.67
query22 3.76 8.18 1.96
query23 18.25 1.40 1.21
query24 2.20 0.22 0.22
query25 0.16 0.08 0.08
query26 0.29 0.22 0.21
query27 0.45 0.23 0.23
query28 13.26 1.02 1.00
query29 12.65 3.31 3.32
query30 0.24 0.07 0.06
query31 2.86 0.40 0.39
query32 3.25 0.47 0.47
query33 2.93 2.88 2.91
query34 17.12 4.32 4.35
query35 4.37 4.43 4.44
query36 0.66 0.46 0.50
query37 0.18 0.15 0.16
query38 0.15 0.14 0.15
query39 0.04 0.04 0.03
query40 0.15 0.11 0.13
query41 0.09 0.05 0.05
query42 0.06 0.05 0.06
query43 0.05 0.04 0.04
Total cold run time: 109.35 s
Total hot run time: 30.73 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1692768644
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -913,320 +1458,420 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
- msg = ss.str();
- return;
- }
- version = version_pb.version();
- } else {
- version = 1;
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
}
- new_versions[version_keys[i]] = version + 1;
+ txn->put(recycle_key, recycle_val);
}
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ code = ret.first;
+ msg = ret.second;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
+ continue;
Review Comment:
Add some comments to explain why we need to retry.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -804,91 +962,483 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
std::stringstream ss;
int64_t txn_id = request->txn_id();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // Get txn info with db_id and txn_id
- std::string info_val; // Will be reused when saving updated txn
- const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
- err = txn->get(info_key, &info_val);
- if (err != TxnErrorCode::TXN_OK) {
- code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
- : cast_as<ErrCategory::READ>(err);
- if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
- } else {
- ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
- << " err=" << err;
+ // Get txn info with db_id and txn_id
+ std::string info_val; // Will be reused when saving updated txn
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
+ } else {
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
+ }
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(info_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // TODO: do more check like txn state, 2PC etc.
- DCHECK(txn_info.txn_id() == txn_id);
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
- code = MetaServiceCode::TXN_ALREADY_ABORTED;
- ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ // TODO: do more check like txn state, 2PC etc.
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
- code = MetaServiceCode::TXN_ALREADY_VISIBLE;
- if (request->has_is_2pc() && request->is_2pc()) {
- ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ if (request->has_is_2pc() && request->is_2pc()) {
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- response->mutable_txn_info()->CopyFrom(txn_info);
- return;
- }
- if (request->has_is_2pc() && request->is_2pc() &&
- txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
- code = MetaServiceCode::TXN_INVALID_STATUS;
- ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Prepare rowset meta and new_versions
+ // Prepare rowset meta and new_versions
+ // Read tablet indexes in batch.
+ std::vector<std::string> tablet_idx_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
+ }
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ size_t total_rowsets = tmp_rowsets_meta.size();
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ for (size_t i = 0; i < total_rowsets; i++) {
+ uint64_t tablet_id = tmp_rowsets_meta[i].second.tablet_id();
+ if (!tablet_idx_values[i].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " tablet_id=" << tablet_id << " key=" << hex(tablet_idx_keys[i]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "malformed tablet index value tablet_id=" << tablet_id
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ VLOG_DEBUG << "tablet_id:" << tablet_id
+ << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ }
+
+ tablet_idx_keys.clear();
+ tablet_idx_values.clear();
+
+ // {table/partition} -> version
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ size_t total_versions = version_keys.size();
+ for (size_t i = 0; i < total_versions; i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+ version_keys.clear();
+ version_values.clear();
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ // std::shared_ptr<TxnLazyCommitTask> task =
+ // txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ // std::pair<MetaServiceCode, std::string> ret = task->wait();
+ // need_advance_last_txn = false;
+ // last_pending_txn_id = 0;
+ // code = ret.first;
+ // msg = ret.second;
+ // if (code != MetaServiceCode::OK) {
+ // LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ // << " code=" << code << "msg=" << msg;
+ // return;
+ // }
+ // continue;
Review Comment:
Missing?
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -913,320 +1458,420 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
- msg = ss.str();
- return;
- }
- version = version_pb.version();
- } else {
- version = 1;
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
}
- new_versions[version_keys[i]] = version + 1;
+ txn->put(recycle_key, recycle_val);
}
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ code = ret.first;
+ msg = ret.second;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
+ continue;
+ }
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ if (request->has_is_2pc() && request->is_2pc()) {
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
-
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
-
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
return;
}
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- // Save table versions
- for (auto& i : table_id_tablet_ids) {
- std::string ver_key = table_version_key({instance_id, db_id, i.first});
- txn->atomic_add(ver_key, 1);
- LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key) << " txn_id=" << txn_id;
- }
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
+ }
- LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
+ // save versions for partition
+ response->clear_table_ids();
+ response->clear_partition_ids();
+ response->clear_versions();
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- // Update txn_info
- txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id;
- auto now_time = system_clock::now();
- uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
- if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
- LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
- << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
- return;
- }
- txn_info.set_commit_time(commit_time);
- txn_info.set_finish_time(commit_time);
- if (request->has_commit_attachment()) {
- txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
- }
- LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
- info_val.clear();
- if (!txn_info.SerializeToString(&info_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(info_key, info_val);
- LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
- // Update stats of affected tablet
- for (auto& [tablet_id, stats] : tablet_stats) {
- DCHECK(tablet_ids.count(tablet_id));
- auto& tablet_idx = tablet_ids[tablet_id];
- StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
- tablet_idx.partition_id(), tablet_id};
- update_tablet_stats(info, stats, txn, code, msg);
- if (code != MetaServiceCode::OK) return;
- }
- // Remove tmp rowset meta
- for (auto& [k, _] : tmp_rowsets_meta) {
- txn->remove(k);
- LOG(INFO) << "xxx remove tmp_rowset_key=" << hex(k) << " txn_id=" << txn_id;
- }
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << "txn_id=" << txn_id << " table_id=" << table_id
+ << " partition_id=" << partition_id << " version=" << i.second;
- const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
- LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
- txn->remove(running_key);
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
- std::string recycle_val;
- std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
- RecycleTxnPB recycle_pb;
- recycle_pb.set_creation_time(commit_time);
- recycle_pb.set_label(txn_info.label());
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- if (!recycle_pb.SerializeToString(&recycle_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(recycle_key, recycle_val);
+ for (size_t i = 0; i < lock_keys.size(); i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- if (txn_info.load_job_source_type() ==
- LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
- put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
- }
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
- LOG(INFO) << "xxx commit_txn put recycle_key key=" << hex(recycle_key) << " txn_id=" << txn_id;
- LOG(INFO) << "commit_txn put_size=" << txn->put_bytes() << " del_size=" << txn->delete_bytes()
- << " num_put_keys=" << txn->num_put_keys() << " num_del_keys=" << txn->num_del_keys()
- << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
- // Finally we are done...
- err = txn->commit();
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::COMMIT>(err);
- ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- return;
- }
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
+ << " txn_id=" << txn_id;
+ }
- // calculate table stats from tablets stats
- std::map<int64_t /*table_id*/, TableStats> table_stats;
- std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
- request->base_tablet_ids().end());
- calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
- for (const auto& pair : table_stats) {
- TableStatsPB* stats_pb = response->add_table_stats();
- auto table_id = pair.first;
- auto stats = pair.second;
- get_pb_from_tablestats(stats, stats_pb);
- stats_pb->set_table_id(table_id);
- VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
- << " table_id=" << table_id
- << " updated_row_count=" << stats_pb->updated_row_count();
- }
+ LOG(INFO) << "commit_txn_eventually put_size=" << txn->put_bytes()
+ << " del_size=" << txn->delete_bytes() << " num_put_keys=" << txn->num_put_keys()
+ << " num_del_keys=" << txn->num_del_keys()
+ << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
- response->mutable_txn_info()->CopyFrom(txn_info);
-} // end commit_txn
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer->submit(instance_id, txn_id);
Review Comment:
In the future, we can pass the already read data to TxnLazyCommitter to optimize read performance. For now, let's focus on merging this PR.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -913,320 +1458,420 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
- msg = ss.str();
- return;
- }
- version = version_pb.version();
- } else {
- version = 1;
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
}
- new_versions[version_keys[i]] = version + 1;
+ txn->put(recycle_key, recycle_val);
}
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ code = ret.first;
+ msg = ret.second;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
+ continue;
+ }
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ if (request->has_is_2pc() && request->is_2pc()) {
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
-
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
-
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
return;
}
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- // Save table versions
- for (auto& i : table_id_tablet_ids) {
- std::string ver_key = table_version_key({instance_id, db_id, i.first});
- txn->atomic_add(ver_key, 1);
- LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key) << " txn_id=" << txn_id;
- }
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
+ }
- LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
+ // save versions for partition
+ response->clear_table_ids();
+ response->clear_partition_ids();
+ response->clear_versions();
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- // Update txn_info
- txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id;
- auto now_time = system_clock::now();
- uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
- if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
- LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
- << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
- return;
- }
- txn_info.set_commit_time(commit_time);
- txn_info.set_finish_time(commit_time);
- if (request->has_commit_attachment()) {
- txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
- }
- LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
- info_val.clear();
- if (!txn_info.SerializeToString(&info_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(info_key, info_val);
- LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
- // Update stats of affected tablet
- for (auto& [tablet_id, stats] : tablet_stats) {
- DCHECK(tablet_ids.count(tablet_id));
- auto& tablet_idx = tablet_ids[tablet_id];
- StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
- tablet_idx.partition_id(), tablet_id};
- update_tablet_stats(info, stats, txn, code, msg);
- if (code != MetaServiceCode::OK) return;
- }
- // Remove tmp rowset meta
- for (auto& [k, _] : tmp_rowsets_meta) {
- txn->remove(k);
- LOG(INFO) << "xxx remove tmp_rowset_key=" << hex(k) << " txn_id=" << txn_id;
- }
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << "txn_id=" << txn_id << " table_id=" << table_id
+ << " partition_id=" << partition_id << " version=" << i.second;
- const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
- LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
- txn->remove(running_key);
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
- std::string recycle_val;
- std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
- RecycleTxnPB recycle_pb;
- recycle_pb.set_creation_time(commit_time);
- recycle_pb.set_label(txn_info.label());
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- if (!recycle_pb.SerializeToString(&recycle_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(recycle_key, recycle_val);
+ for (size_t i = 0; i < lock_keys.size(); i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- if (txn_info.load_job_source_type() ==
- LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
- put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
- }
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
- LOG(INFO) << "xxx commit_txn put recycle_key key=" << hex(recycle_key) << " txn_id=" << txn_id;
- LOG(INFO) << "commit_txn put_size=" << txn->put_bytes() << " del_size=" << txn->delete_bytes()
- << " num_put_keys=" << txn->num_put_keys() << " num_del_keys=" << txn->num_del_keys()
- << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
- // Finally we are done...
- err = txn->commit();
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::COMMIT>(err);
- ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- return;
- }
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
+ << " txn_id=" << txn_id;
+ }
- // calculate table stats from tablets stats
- std::map<int64_t /*table_id*/, TableStats> table_stats;
- std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
- request->base_tablet_ids().end());
- calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
- for (const auto& pair : table_stats) {
- TableStatsPB* stats_pb = response->add_table_stats();
- auto table_id = pair.first;
- auto stats = pair.second;
- get_pb_from_tablestats(stats, stats_pb);
- stats_pb->set_table_id(table_id);
- VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
- << " table_id=" << table_id
- << " updated_row_count=" << stats_pb->updated_row_count();
- }
+ LOG(INFO) << "commit_txn_eventually put_size=" << txn->put_bytes()
+ << " del_size=" << txn->delete_bytes() << " num_put_keys=" << txn->num_put_keys()
+ << " num_del_keys=" << txn->num_del_keys()
+ << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
- response->mutable_txn_info()->CopyFrom(txn_info);
-} // end commit_txn
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer->submit(instance_id, txn_id);
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
Review Comment:
```suggestion
std::tie(code, msg) = task->wait();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2260705985
<details>
<summary>TPC-DS: <b>Total hot run time: 170002 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f23de131c772e5e5b480379dddd6cae001bf04a6, data reload: false
query1 923 373 372 372
query2 6466 1714 1613 1613
query3 6652 215 221 215
query4 20217 17552 17284 17284
query5 3642 528 520 520
query6 310 168 187 168
query7 4595 298 302 298
query8 243 202 190 190
query9 8512 2403 2374 2374
query10 450 279 284 279
query11 10457 10081 10106 10081
query12 119 88 92 88
query13 1616 382 370 370
query14 9005 7054 8274 7054
query15 205 171 161 161
query16 6864 444 455 444
query17 954 570 563 563
query18 1808 296 304 296
query19 195 150 147 147
query20 95 89 89 89
query21 208 102 103 102
query22 4135 4083 4048 4048
query23 33793 33840 33649 33649
query24 10461 3126 3040 3040
query25 687 421 411 411
query26 1789 164 160 160
query27 3149 289 282 282
query28 7741 2047 2033 2033
query29 1292 445 449 445
query30 237 162 170 162
query31 955 767 758 758
query32 115 59 56 56
query33 686 323 359 323
query34 935 482 504 482
query35 882 803 764 764
query36 1046 863 898 863
query37 283 96 82 82
query38 2905 2816 2806 2806
query39 860 830 822 822
query40 264 116 117 116
query41 47 46 44 44
query42 135 102 99 99
query43 458 422 400 400
query44 1195 737 736 736
query45 214 180 180 180
query46 1086 795 776 776
query47 1808 1713 1696 1696
query48 365 291 296 291
query49 931 415 418 415
query50 914 439 433 433
query51 6697 6738 6608 6608
query52 109 87 96 87
query53 251 182 178 178
query54 650 461 452 452
query55 78 75 77 75
query56 286 256 248 248
query57 1127 1000 1073 1000
query58 281 263 261 261
query59 2623 2309 2342 2309
query60 301 281 283 281
query61 98 100 96 96
query62 886 656 667 656
query63 207 182 187 182
query64 5673 1968 1980 1968
query65 3172 3102 3103 3102
query66 1351 354 331 331
query67 15497 14829 14849 14829
query68 4332 583 571 571
query69 642 386 315 315
query70 1069 1004 1038 1004
query71 443 284 275 275
query72 7099 2736 2483 2483
query73 778 325 325 325
query74 5992 5678 5585 5585
query75 3505 2795 2776 2776
query76 2762 1322 1410 1322
query77 565 305 306 305
query78 9477 8868 8959 8868
query79 1914 555 544 544
query80 952 508 499 499
query81 576 229 225 225
query82 1025 134 128 128
query83 249 171 173 171
query84 278 82 94 82
query85 1359 320 307 307
query86 444 297 285 285
query87 3261 3133 3130 3130
query88 2968 2430 2418 2418
query89 392 298 287 287
query90 1738 194 193 193
query91 126 102 103 102
query92 63 50 53 50
query93 1758 632 640 632
query94 828 298 305 298
query95 392 279 275 275
query96 610 279 278 278
query97 3236 3047 3098 3047
query98 220 208 197 197
query99 1612 1282 1310 1282
Total cold run time: 262856 ms
Total hot run time: 170002 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2262300969
<details>
<summary>TPC-H: <b>Total hot run time: 41825 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f40d3af909e3a4ce446ee5217827a6811bce34d1, data reload: false
------ Round 1 ----------------------------------
q1 17655 4407 4071 4071
q2 2028 208 202 202
q3 10546 1379 1373 1373
q4 10237 818 935 818
q5 7634 3022 2976 2976
q6 219 135 140 135
q7 1039 612 606 606
q8 9443 1881 1988 1881
q9 8530 6629 6621 6621
q10 8797 3904 3892 3892
q11 435 249 239 239
q12 412 230 225 225
q13 17774 2969 2945 2945
q14 269 250 247 247
q15 517 479 484 479
q16 518 380 393 380
q17 998 889 918 889
q18 8073 7286 7355 7286
q19 1490 1222 1220 1220
q20 549 341 342 341
q21 5317 4724 4798 4724
q22 349 282 275 275
Total cold run time: 112829 ms
Total hot run time: 41825 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4136 4032 4002 4002
q2 324 218 216 216
q3 2992 3044 3185 3044
q4 2028 2039 2006 2006
q5 5679 5470 5473 5470
q6 220 130 130 130
q7 2187 1859 1819 1819
q8 3341 3390 3372 3372
q9 8688 8711 8782 8711
q10 3962 4136 3933 3933
q11 546 455 476 455
q12 781 601 580 580
q13 16313 3157 3132 3132
q14 299 277 276 276
q15 530 497 487 487
q16 470 415 440 415
q17 1755 1748 1737 1737
q18 8350 7779 7726 7726
q19 1740 1707 1710 1707
q20 2054 1894 1828 1828
q21 5857 5468 5419 5419
q22 531 450 475 450
Total cold run time: 72783 ms
Total hot run time: 56915 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312706402
<details>
<summary>ClickBench: <b>Total hot run time: 31.14 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5f3b5ea8bfb6214887e8d1bb42d8f87233ccc8a0, data reload: false
query1 0.05 0.05 0.04
query2 0.07 0.04 0.04
query3 0.23 0.05 0.05
query4 1.67 0.06 0.07
query5 0.52 0.48 0.49
query6 1.13 0.72 0.72
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.56 0.49 0.48
query10 0.53 0.55 0.56
query11 0.16 0.12 0.12
query12 0.15 0.12 0.12
query13 0.62 0.60 0.59
query14 0.76 0.79 0.77
query15 0.84 0.82 0.82
query16 0.39 0.37 0.37
query17 1.04 1.03 0.96
query18 0.22 0.21 0.20
query19 1.95 1.79 1.72
query20 0.02 0.00 0.01
query21 15.40 0.64 0.64
query22 4.17 7.19 2.44
query23 18.32 1.40 1.31
query24 2.06 0.22 0.22
query25 0.16 0.08 0.08
query26 0.27 0.17 0.18
query27 0.07 0.07 0.07
query28 13.24 1.01 1.00
query29 12.56 3.42 3.32
query30 0.25 0.06 0.05
query31 2.86 0.39 0.38
query32 3.28 0.48 0.47
query33 2.92 3.02 2.98
query34 16.76 4.38 4.36
query35 4.41 4.43 4.49
query36 0.66 0.49 0.48
query37 0.19 0.16 0.15
query38 0.15 0.14 0.15
query39 0.05 0.04 0.04
query40 0.16 0.13 0.13
query41 0.09 0.05 0.05
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 109.12 s
Total hot run time: 31.14 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2304123512
<details>
<summary>TPC-H: <b>Total hot run time: 38205 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 20e25bf51ae7f01a6fff2f9704c155a8c059045a, data reload: false
------ Round 1 ----------------------------------
q1 19192 4702 4345 4345
q2 2041 194 177 177
q3 10529 1143 1093 1093
q4 10231 818 702 702
q5 7742 2906 2785 2785
q6 226 139 143 139
q7 970 612 594 594
q8 9320 2070 2070 2070
q9 7204 6550 6517 6517
q10 7024 2200 2236 2200
q11 450 255 247 247
q12 397 225 230 225
q13 17764 3034 3046 3034
q14 282 232 234 232
q15 525 487 507 487
q16 502 385 385 385
q17 992 664 772 664
q18 7521 6772 6869 6772
q19 1403 1091 1011 1011
q20 671 346 337 337
q21 3893 3193 3204 3193
q22 1125 1017 996 996
Total cold run time: 110004 ms
Total hot run time: 38205 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4424 4320 4270 4270
q2 383 275 275 275
q3 2903 2654 2668 2654
q4 1962 1681 1669 1669
q5 5417 5375 5403 5375
q6 219 132 135 132
q7 2111 1735 1750 1735
q8 3193 3374 3346 3346
q9 8419 8461 8505 8461
q10 3449 3180 3191 3180
q11 626 487 495 487
q12 779 627 589 589
q13 14709 3061 3015 3015
q14 309 277 284 277
q15 516 497 470 470
q16 469 434 427 427
q17 1792 1498 1503 1498
q18 7758 7556 7379 7379
q19 1677 1584 1601 1584
q20 2058 1821 1855 1821
q21 5517 5257 5090 5090
q22 1095 1045 1011 1011
Total cold run time: 69785 ms
Total hot run time: 54745 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2249642741
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2245067398
<details>
<summary>TPC-H: <b>Total hot run time: 39622 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d47f33a15e68079183da96a15302b529dc1c36a2, data reload: false
------ Round 1 ----------------------------------
q1 17610 4591 4263 4263
q2 2019 191 183 183
q3 10556 1161 1064 1064
q4 10294 769 821 769
q5 7575 2708 2662 2662
q6 222 139 134 134
q7 957 606 591 591
q8 9220 2038 2078 2038
q9 8838 6519 6549 6519
q10 8687 3803 3783 3783
q11 454 230 238 230
q12 410 231 223 223
q13 17772 2996 3008 2996
q14 280 239 251 239
q15 519 484 487 484
q16 495 393 382 382
q17 964 687 677 677
q18 8151 7595 7343 7343
q19 5829 1332 1256 1256
q20 677 328 319 319
q21 4912 3178 4004 3178
q22 354 289 294 289
Total cold run time: 116795 ms
Total hot run time: 39622 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4369 4246 4227 4227
q2 363 270 262 262
q3 2996 2880 2880 2880
q4 2022 1704 1774 1704
q5 5665 5532 5524 5524
q6 220 135 131 131
q7 2212 1865 1850 1850
q8 3276 3410 3423 3410
q9 8768 8788 8888 8788
q10 4095 3860 3791 3791
q11 581 499 493 493
q12 819 644 645 644
q13 15995 3187 3186 3186
q14 325 291 287 287
q15 530 491 477 477
q16 502 427 437 427
q17 1829 1516 1523 1516
q18 8112 7917 7832 7832
q19 1714 1663 1480 1480
q20 2138 1909 1857 1857
q21 5088 4874 4896 4874
q22 586 507 527 507
Total cold run time: 72205 ms
Total hot run time: 56147 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1708897290
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,371 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:208:** 122 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
##########
cloud/test/txn_lazy_commit_test.cpp:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <brpc/controller.h>
Review Comment:
warning: 'brpc/controller.h' file not found [clang-diagnostic-error]
```cpp
#include <brpc/controller.h>
^
```
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,371 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
Review Comment:
warning: function 'convert_tmp_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void convert_tmp_rowsets(
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:36:** 152 lines including whitespace and comments (threshold 80)
```cpp
void convert_tmp_rowsets(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1708714602
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1888,72 +2019,90 @@ int InstanceRecycler::abort_timeout_txn() {
}
++num_timeout;
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv_->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- LOG_ERROR("failed to create txn err={}", err).tag("key", hex(k));
- return -1;
- }
- std::string_view k1 = k;
- //TxnRunningKeyInfo 0:instance_id 1:db_id 2:txn_id
- k1.remove_prefix(1); // Remove key space
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- if (decode_key(&k1, &out) != 0) {
- LOG_ERROR("failed to decode key").tag("key", hex(k));
- return -1;
- }
- int64_t db_id = std::get<int64_t>(std::get<0>(out[3]));
- int64_t txn_id = std::get<int64_t>(std::get<0>(out[4]));
- VLOG_DEBUG << "instance_id=" << instance_id_ << " db_id=" << db_id << " txn_id=" << txn_id;
- // Update txn_info
- std::string txn_inf_key, txn_inf_val;
- txn_info_key({instance_id_, db_id, txn_id}, &txn_inf_key);
- err = txn->get(txn_inf_key, &txn_inf_val);
- if (err != TxnErrorCode::TXN_OK) {
- LOG_WARNING("failed to get txn info err={}", err).tag("key", hex(txn_inf_key));
- return -1;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(txn_inf_val)) {
- LOG_WARNING("failed to parse txn info").tag("key", hex(k));
- return -1;
- }
- txn_info.set_status(TxnStatusPB::TXN_STATUS_ABORTED);
- txn_info.set_finish_time(current_time);
- txn_info.set_reason("timeout");
- VLOG_DEBUG << "txn_info=" << txn_info.DebugString();
- txn_inf_val.clear();
- if (!txn_info.SerializeToString(&txn_inf_val)) {
- LOG_WARNING("failed to serialize txn info").tag("key", hex(k));
- return -1;
- }
- txn->put(txn_inf_key, txn_inf_val);
- VLOG_DEBUG << "txn->put, txn_inf_key=" << hex(txn_inf_key);
- // Put recycle txn key
- std::string recyc_txn_key, recyc_txn_val;
- recycle_txn_key({instance_id_, db_id, txn_id}, &recyc_txn_key);
- RecycleTxnPB recycle_txn_pb;
- recycle_txn_pb.set_creation_time(current_time);
- recycle_txn_pb.set_label(txn_info.label());
- if (!recycle_txn_pb.SerializeToString(&recyc_txn_val)) {
- LOG_WARNING("failed to serialize txn recycle info")
- .tag("key", hex(k))
- .tag("db_id", db_id)
- .tag("txn_id", txn_id);
- return -1;
- }
- txn->put(recyc_txn_key, recyc_txn_val);
- // Remove txn running key
- txn->remove(k);
- err = txn->commit();
- if (err != TxnErrorCode::TXN_OK) {
- LOG_WARNING("failed to commit txn err={}", err)
- .tag("key", hex(k))
- .tag("db_id", db_id)
- .tag("txn_id", txn_id);
- return -1;
- }
- ++num_abort;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ LOG_ERROR("failed to create txn err={}", err).tag("key", hex(k));
+ return -1;
+ }
+ std::string_view k1 = k;
+ //TxnRunningKeyInfo 0:instance_id 1:db_id 2:txn_id
+ k1.remove_prefix(1); // Remove key space
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ if (decode_key(&k1, &out) != 0) {
+ LOG_ERROR("failed to decode key").tag("key", hex(k));
+ return -1;
+ }
+ int64_t db_id = std::get<int64_t>(std::get<0>(out[3]));
+ int64_t txn_id = std::get<int64_t>(std::get<0>(out[4]));
+ VLOG_DEBUG << "instance_id=" << instance_id_ << " db_id=" << db_id
+ << " txn_id=" << txn_id;
+ // Update txn_info
+ std::string txn_inf_key, txn_inf_val;
+ txn_info_key({instance_id_, db_id, txn_id}, &txn_inf_key);
+ err = txn->get(txn_inf_key, &txn_inf_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ LOG_WARNING("failed to get txn info err={}", err).tag("key", hex(txn_inf_key));
+ return -1;
+ }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(txn_inf_val)) {
+ LOG_WARNING("failed to parse txn info").tag("key", hex(k));
+ return -1;
+ }
+
+ if (TxnStatusPB::TXN_STATUS_COMMITTED == txn_info.status()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id_, txn_info.txn_id());
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ if (ret.first != MetaServiceCode::OK) {
+ LOG(WARNING) << "lazy commit txn failed txn_id=" << txn_id
+ << " code=" << ret.first << "msg=" << ret.second;
+ return -1;
+ }
+ continue;
+ } else {
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_ABORTED);
Review Comment:
The `continue` means retry to recycle this "timeouted" TXN, but it has been committed (status visible) via the `TxnLazyCommitTask`, then this DCHECK will panic.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -788,6 +788,164 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
txn->put(key, val);
}
}
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
Review Comment:
`convert_tmp_rowsets` seems only used in TxnLazyCommiter, and line 986 is `DCHECK(partition_versions.size() == 1);`.
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -326,8 +342,12 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
std::vector<std::optional<std::string>> version_values;
version_keys.reserve(BATCH_SIZE);
version_values.reserve(BATCH_SIZE);
+
while ((code == MetaServiceCode::OK || code == MetaServiceCode::KV_TXN_TOO_OLD) &&
response->versions_size() < response->partition_ids_size()) {
+ response->clear_versions();
+ code = MetaServiceCode::OK;
+
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2259578842
<details>
<summary>TPC-H: <b>Total hot run time: 41912 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ed44221f04cb0a14f7132c207fbd3cf93e572f1, data reload: false
------ Round 1 ----------------------------------
q1 17781 4217 4081 4081
q2 2028 204 203 203
q3 10544 1343 1350 1343
q4 10233 886 939 886
q5 7670 2956 3002 2956
q6 222 138 138 138
q7 1047 617 626 617
q8 9452 1953 1979 1953
q9 8581 6628 6655 6628
q10 8755 3878 3841 3841
q11 426 258 253 253
q12 409 226 232 226
q13 17766 2947 2972 2947
q14 274 243 247 243
q15 530 487 495 487
q16 506 407 393 393
q17 977 948 911 911
q18 8058 7242 7300 7242
q19 1437 1226 1221 1221
q20 560 334 343 334
q21 5293 4729 4901 4729
q22 357 287 280 280
Total cold run time: 112906 ms
Total hot run time: 41912 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4042 4026 4059 4026
q2 336 231 237 231
q3 3023 3063 3167 3063
q4 2029 2006 1987 1987
q5 5680 5501 5509 5501
q6 218 133 132 132
q7 2188 1843 1839 1839
q8 3394 3455 3400 3400
q9 8734 8763 8839 8763
q10 3948 4064 3935 3935
q11 557 453 465 453
q12 761 610 602 602
q13 15136 3184 3162 3162
q14 308 275 288 275
q15 533 501 488 488
q16 500 416 415 415
q17 1770 1757 1754 1754
q18 8415 7710 7738 7710
q19 1766 1746 1729 1729
q20 2068 1861 1841 1841
q21 5836 5525 5278 5278
q22 537 467 469 467
Total cold run time: 71779 ms
Total hot run time: 57051 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2283623317
<details>
<summary>TPC-H: <b>Total hot run time: 39992 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 88d3d52a5127df11de69d64be57d1e49cab899b7, data reload: false
------ Round 1 ----------------------------------
q1 18613 4819 4379 4379
q2 2629 189 191 189
q3 11330 1169 1177 1169
q4 10588 760 732 732
q5 7616 2585 2580 2580
q6 231 139 145 139
q7 989 628 614 614
q8 9927 1934 1946 1934
q9 8769 6582 6620 6582
q10 7058 2190 2210 2190
q11 476 248 247 247
q12 401 218 212 212
q13 17847 2991 2961 2961
q14 276 242 239 239
q15 543 479 496 479
q16 640 532 514 514
q17 995 651 667 651
q18 8178 7472 7455 7455
q19 4377 1072 980 980
q20 743 421 413 413
q21 5404 4693 4319 4319
q22 1103 1058 1014 1014
Total cold run time: 118733 ms
Total hot run time: 39992 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4473 4231 4294 4231
q2 392 288 268 268
q3 2835 2650 2664 2650
q4 1876 1639 1588 1588
q5 5259 5275 5286 5275
q6 223 129 129 129
q7 2071 1652 1653 1652
q8 3133 3314 3349 3314
q9 8362 8329 8349 8329
q10 3372 3116 3118 3116
q11 584 490 487 487
q12 753 617 592 592
q13 17360 3000 2970 2970
q14 300 274 284 274
q15 532 476 482 476
q16 608 548 558 548
q17 1807 1493 1479 1479
q18 7627 7485 7457 7457
q19 1651 1625 1566 1566
q20 2060 1888 1886 1886
q21 5303 5005 5053 5005
q22 1086 1036 1016 1016
Total cold run time: 71667 ms
Total hot run time: 54308 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1715102836
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1707,6 +1777,62 @@ int InstanceRecycler::recycle_rowsets() {
return ret;
}
+bool check_txn_abort(std::shared_ptr<TxnKv> txn_kv, const std::string& instance_id,
+ int64_t txn_id) {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ LOG(WARNING) << "failed to create txn, txn_id=" << txn_id << " instance_id=" << instance_id;
+ return false;
+ }
+
+ std::string index_val;
+ const std::string index_key = txn_index_key({instance_id, txn_id});
+ err = txn->get(index_key, &index_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // txn has been recycled;
+ return true;
+ }
+ LOG(WARNING) << "failed to get txn index key, txn_id=" << txn_id
+ << " instance_id=" << instance_id << " key=" << hex(index_key);
+ return false;
+ }
+
+ TxnIndexPB index_pb;
+ if (!index_pb.ParseFromString(index_val)) {
+ LOG(WARNING) << "failed to parse txn_index_pb, txn_id=" << txn_id
+ << " instance_id=" << instance_id;
+ return false;
+ }
+
+ DCHECK(index_pb.has_tablet_index() == true);
+ DCHECK(index_pb.tablet_index().has_db_id() == true);
+ int64_t db_id = index_pb.tablet_index().db_id();
Review Comment:
For compatibility, consider checking the DB id before using it, just like:
```c++
if (!index_pb.tablet_index().has_db_id()) {
return false;
}
```
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1707,6 +1777,62 @@ int InstanceRecycler::recycle_rowsets() {
return ret;
}
+bool check_txn_abort(std::shared_ptr<TxnKv> txn_kv, const std::string& instance_id,
+ int64_t txn_id) {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ LOG(WARNING) << "failed to create txn, txn_id=" << txn_id << " instance_id=" << instance_id;
+ return false;
+ }
+
+ std::string index_val;
+ const std::string index_key = txn_index_key({instance_id, txn_id});
+ err = txn->get(index_key, &index_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // txn has been recycled;
+ return true;
+ }
+ LOG(WARNING) << "failed to get txn index key, txn_id=" << txn_id
+ << " instance_id=" << instance_id << " key=" << hex(index_key);
+ return false;
+ }
+
+ TxnIndexPB index_pb;
+ if (!index_pb.ParseFromString(index_val)) {
+ LOG(WARNING) << "failed to parse txn_index_pb, txn_id=" << txn_id
+ << " instance_id=" << instance_id;
+ return false;
+ }
+
+ DCHECK(index_pb.has_tablet_index() == true);
+ DCHECK(index_pb.tablet_index().has_db_id() == true);
+ int64_t db_id = index_pb.tablet_index().db_id();
Review Comment:
A new kind of task needs to be added. This task should scan all partition keys, parse db_id, and fill the db_id of the `TabletIndexPB`.
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1918,42 +2043,69 @@ int InstanceRecycler::abort_timeout_txn() {
LOG_WARNING("failed to parse txn info").tag("key", hex(k));
return -1;
}
- txn_info.set_status(TxnStatusPB::TXN_STATUS_ABORTED);
- txn_info.set_finish_time(current_time);
- txn_info.set_reason("timeout");
- VLOG_DEBUG << "txn_info=" << txn_info.DebugString();
- txn_inf_val.clear();
- if (!txn_info.SerializeToString(&txn_inf_val)) {
- LOG_WARNING("failed to serialize txn info").tag("key", hex(k));
- return -1;
- }
- txn->put(txn_inf_key, txn_inf_val);
- VLOG_DEBUG << "txn->put, txn_inf_key=" << hex(txn_inf_key);
- // Put recycle txn key
- std::string recyc_txn_key, recyc_txn_val;
- recycle_txn_key({instance_id_, db_id, txn_id}, &recyc_txn_key);
- RecycleTxnPB recycle_txn_pb;
- recycle_txn_pb.set_creation_time(current_time);
- recycle_txn_pb.set_label(txn_info.label());
- if (!recycle_txn_pb.SerializeToString(&recyc_txn_val)) {
- LOG_WARNING("failed to serialize txn recycle info")
- .tag("key", hex(k))
- .tag("db_id", db_id)
- .tag("txn_id", txn_id);
- return -1;
- }
- txn->put(recyc_txn_key, recyc_txn_val);
- // Remove txn running key
- txn->remove(k);
- err = txn->commit();
- if (err != TxnErrorCode::TXN_OK) {
- LOG_WARNING("failed to commit txn err={}", err)
- .tag("key", hex(k))
- .tag("db_id", db_id)
- .tag("txn_id", txn_id);
- return -1;
+
+ if (TxnStatusPB::TXN_STATUS_COMMITTED == txn_info.status()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id_, txn_info.txn_id());
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ if (ret.first != MetaServiceCode::OK) {
+ LOG(WARNING) << "lazy commit txn failed txn_id=" << txn_id << " code=" << ret.first
+ << "msg=" << ret.second;
+ return -1;
+ }
+ ++num_advance;
+ return 0;
+ } else {
Review Comment:
A suggestion, remove this `else {`, so the diff view will be clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1722773609
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -913,320 +1458,420 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
- msg = ss.str();
- return;
- }
- version = version_pb.version();
- } else {
- version = 1;
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
}
- new_versions[version_keys[i]] = version + 1;
+ txn->put(recycle_key, recycle_val);
}
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ code = ret.first;
+ msg = ret.second;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
+ continue;
Review Comment:
> Add some comments to explain why we need to retry.
done, i have added a comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2303598622
run cloudut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1708566789
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -237,46 +238,63 @@ void MetaServiceImpl::get_version(::google::protobuf::RpcController* controller,
partition_version_key({instance_id, db_id, table_id, partition_id}, &ver_key);
}
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv_->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- msg = "failed to create txn";
- code = cast_as<ErrCategory::CREATE>(err);
- return;
- }
+ do {
+ code = MetaServiceCode::OK;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ msg = "failed to create txn";
+ code = cast_as<ErrCategory::CREATE>(err);
+ return;
+ }
- std::string ver_val;
- // 0 for success get a key, 1 for key not found, negative for error
- err = txn->get(ver_key, &ver_val);
- VLOG_DEBUG << "xxx get version_key=" << hex(ver_key);
- if (err == TxnErrorCode::TXN_OK) {
- if (is_table_version) {
- int64_t version = 0;
- if (!txn->decode_atomic_int(ver_val, &version)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "malformed table version value";
- return;
- }
- response->set_version(version);
- } else {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(ver_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "malformed version value";
- return;
+ std::string ver_val;
+ // 0 for success get a key, 1 for key not found, negative for error
+ err = txn->get(ver_key, &ver_val);
+ VLOG_DEBUG << "xxx get version_key=" << hex(ver_key);
+ if (err == TxnErrorCode::TXN_OK) {
+ if (is_table_version) {
+ int64_t version = 0;
+ if (!txn->decode_atomic_int(ver_val, &version)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "malformed table version value";
+ return;
+ }
+ response->set_version(version);
+ } else {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "malformed version value";
+ return;
+ }
+
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
+
+ std::tie(code, msg) = task->wait();
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING)
+ << "wait txn lazy commit failed, txn_id=" << version_pb.txn_id();
+ return;
+ }
+ }
+
+ response->set_version(version_pb.version());
+ response->add_version_update_time_ms(version_pb.update_time_ms());
}
- response->set_version(version_pb.version());
- response->add_version_update_time_ms(version_pb.update_time_ms());
+ { TEST_SYNC_POINT_CALLBACK("get_version_code", &code); }
+ return;
+ } else if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ msg = "not found";
+ code = MetaServiceCode::VERSION_NOT_FOUND;
+ return;
}
- { TEST_SYNC_POINT_CALLBACK("get_version_code", &code); }
- return;
- } else if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- msg = "not found";
- code = MetaServiceCode::VERSION_NOT_FOUND;
- return;
- }
- msg = fmt::format("failed to get txn, err={}", err);
- code = cast_as<ErrCategory::READ>(err);
+ msg = fmt::format("failed to get txn, err={}", err);
+ code = cast_as<ErrCategory::READ>(err);
+ } while (false);
Review Comment:
> remove the `while`, since the `continue` are removed.
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1708573142
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:51:** 122 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2304148055
<details>
<summary>TPC-DS: <b>Total hot run time: 187877 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 20e25bf51ae7f01a6fff2f9704c155a8c059045a, data reload: false
query1 912 367 370 367
query2 6460 2003 1946 1946
query3 6646 209 215 209
query4 34307 23218 23202 23202
query5 4138 517 486 486
query6 258 165 157 157
query7 4596 298 296 296
query8 266 213 214 213
query9 8471 2436 2458 2436
query10 438 285 267 267
query11 17646 15092 15030 15030
query12 158 104 96 96
query13 1628 379 367 367
query14 10029 7431 7436 7431
query15 280 174 179 174
query16 8105 462 493 462
query17 1631 592 568 568
query18 2144 303 297 297
query19 338 154 156 154
query20 118 112 108 108
query21 211 107 103 103
query22 4503 4246 4237 4237
query23 34461 33369 33427 33369
query24 11101 2880 2892 2880
query25 656 410 402 402
query26 1405 166 167 166
query27 2811 288 286 286
query28 7129 2040 2033 2033
query29 907 432 433 432
query30 318 169 152 152
query31 989 783 756 756
query32 100 64 64 64
query33 771 299 303 299
query34 955 492 501 492
query35 869 746 719 719
query36 1111 938 950 938
query37 158 88 87 87
query38 3975 3880 3824 3824
query39 1454 1440 1380 1380
query40 274 123 120 120
query41 49 49 52 49
query42 124 107 99 99
query43 522 453 458 453
query44 1183 756 780 756
query45 198 168 170 168
query46 1117 781 750 750
query47 1876 1776 1796 1776
query48 382 307 298 298
query49 1064 437 455 437
query50 823 433 428 428
query51 7200 7009 7031 7009
query52 101 93 93 93
query53 262 189 186 186
query54 1064 467 479 467
query55 82 78 80 78
query56 302 259 267 259
query57 1195 1091 1086 1086
query58 268 229 266 229
query59 2939 2838 2747 2747
query60 314 282 291 282
query61 126 124 128 124
query62 862 670 677 670
query63 232 192 198 192
query64 6619 2326 1740 1740
query65 3258 3261 3168 3168
query66 1365 348 390 348
query67 15659 15165 15311 15165
query68 4826 575 570 570
query69 641 369 298 298
query70 1181 1083 1133 1083
query71 448 285 303 285
query72 7274 2306 2079 2079
query73 785 333 346 333
query74 9276 8868 8766 8766
query75 3742 2708 2746 2708
query76 3163 1016 983 983
query77 679 331 331 331
query78 9843 8989 9581 8989
query79 2803 545 548 545
query80 831 505 519 505
query81 587 233 229 229
query82 1192 142 133 133
query83 287 153 153 153
query84 239 77 78 77
query85 1556 285 283 283
query86 337 304 294 294
query87 4407 4243 4399 4243
query88 3986 2396 2369 2369
query89 399 290 287 287
query90 1638 203 202 202
query91 181 102 101 101
query92 61 58 54 54
query93 2196 567 558 558
query94 616 303 318 303
query95 343 270 264 264
query96 641 275 275 275
query97 3197 3084 3043 3043
query98 231 202 206 202
query99 1731 1270 1290 1270
Total cold run time: 301230 ms
Total hot run time: 187877 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2302343162
<details>
<summary>TPC-DS: <b>Total hot run time: 199083 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ecb308aa0ea6ae75918b441fcce5ed28f71c479, data reload: false
query1 1327 922 876 876
query2 6544 1987 1894 1894
query3 10663 4057 4004 4004
query4 59596 27013 23328 23328
query5 5505 687 708 687
query6 454 214 212 212
query7 5801 328 333 328
query8 521 437 440 437
query9 8947 2561 2545 2545
query10 559 349 351 349
query11 16591 15251 15522 15251
query12 205 145 144 144
query13 1636 497 451 451
query14 11528 8031 7960 7960
query15 251 199 193 193
query16 7039 462 501 462
query17 1177 641 631 631
query18 1965 375 346 346
query19 310 172 179 172
query20 160 143 137 137
query21 253 147 147 147
query22 4677 4472 4433 4433
query23 34993 33978 34414 33978
query24 6218 3056 3047 3047
query25 579 433 441 433
query26 705 184 184 184
query27 1708 315 329 315
query28 3623 2184 2175 2175
query29 714 458 457 457
query30 243 206 197 197
query31 1036 887 895 887
query32 110 80 90 80
query33 535 368 369 368
query34 932 515 527 515
query35 880 759 753 753
query36 1130 949 996 949
query37 171 115 112 112
query38 4013 3936 3821 3821
query39 1538 1499 1492 1492
query40 241 160 156 156
query41 139 136 140 136
query42 137 121 115 115
query43 548 520 508 508
query44 1137 801 802 801
query45 224 193 193 193
query46 1117 799 798 798
query47 1966 1863 1842 1842
query48 416 348 347 347
query49 920 601 604 601
query50 912 478 485 478
query51 7343 7126 7043 7043
query52 122 111 111 111
query53 304 230 232 230
query54 647 524 520 520
query55 94 96 94 94
query56 349 339 332 332
query57 1247 1157 1152 1152
query58 298 322 332 322
query59 2996 2754 2791 2754
query60 367 334 340 334
query61 160 152 152 152
query62 801 682 761 682
query63 267 238 239 238
query64 3044 1899 1871 1871
query65 3283 3227 3234 3227
query66 960 679 693 679
query67 15472 15357 15298 15298
query68 4511 616 602 602
query69 472 334 333 333
query70 1242 1170 1139 1139
query71 384 324 328 324
query72 6440 2325 2113 2113
query73 800 368 376 368
query74 9060 8864 8879 8864
query75 3433 2704 2796 2704
query76 1773 1088 1080 1080
query77 674 446 466 446
query78 9879 9232 9017 9017
query79 1136 582 574 574
query80 978 630 630 630
query81 577 262 271 262
query82 861 167 176 167
query83 239 216 268 216
query84 289 97 99 97
query85 853 366 367 366
query86 331 339 324 324
query87 4473 4333 4236 4236
query88 3444 2550 2551 2550
query89 434 348 324 324
query90 1809 242 238 238
query91 159 133 135 133
query92 96 78 77 77
query93 1424 561 557 557
query94 700 333 331 331
query95 398 305 314 305
query96 615 304 285 285
query97 3221 3127 3116 3116
query98 239 227 223 223
query99 1871 1318 1318 1318
Total cold run time: 313163 ms
Total hot run time: 199083 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2314312023
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2314380356
<details>
<summary>TPC-DS: <b>Total hot run time: 195283 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cbbeda91329c7f9ecaf9f55ca0b98af79b86d7ee, data reload: false
query1 1271 898 869 869
query2 6287 2025 2028 2025
query3 10783 4136 4143 4136
query4 60232 26807 23447 23447
query5 5309 500 499 499
query6 411 162 163 162
query7 5754 300 300 300
query8 316 218 218 218
query9 8643 2501 2480 2480
query10 466 269 273 269
query11 18168 15119 15538 15119
query12 160 105 102 102
query13 1502 403 403 403
query14 10627 7806 7516 7516
query15 225 182 193 182
query16 7262 500 473 473
query17 1102 593 611 593
query18 1981 311 312 311
query19 291 153 154 153
query20 122 119 123 119
query21 212 112 113 112
query22 4733 4626 4523 4523
query23 34472 33691 33805 33691
query24 6008 2936 2862 2862
query25 520 407 401 401
query26 649 165 163 163
query27 1712 279 282 279
query28 3942 2146 2142 2142
query29 678 428 429 428
query30 243 162 163 162
query31 936 771 769 769
query32 81 60 61 60
query33 428 292 293 292
query34 873 486 488 486
query35 833 722 766 722
query36 1058 958 977 958
query37 152 93 94 93
query38 3962 3815 4024 3815
query39 1460 1377 1395 1377
query40 203 123 118 118
query41 49 47 46 46
query42 114 99 101 99
query43 511 487 491 487
query44 1085 748 751 748
query45 199 167 168 167
query46 1096 773 763 763
query47 1866 1828 1802 1802
query48 385 300 304 300
query49 783 440 445 440
query50 826 440 420 420
query51 7263 7137 7115 7115
query52 97 89 90 89
query53 259 184 188 184
query54 577 478 473 473
query55 77 81 83 81
query56 279 267 270 267
query57 1192 1109 1080 1080
query58 233 250 228 228
query59 3129 2899 3049 2899
query60 296 266 275 266
query61 104 94 92 92
query62 768 664 635 635
query63 211 182 181 181
query64 2471 675 663 663
query65 3236 3161 3194 3161
query66 657 333 367 333
query67 15576 15299 15303 15299
query68 4378 571 560 560
query69 528 372 299 299
query70 1153 1067 1063 1063
query71 400 279 281 279
query72 6470 4107 4002 4002
query73 778 337 334 334
query74 9234 8848 9004 8848
query75 3389 2725 2754 2725
query76 1682 1036 1060 1036
query77 648 321 318 318
query78 11039 9489 9257 9257
query79 1976 537 547 537
query80 813 534 501 501
query81 567 236 242 236
query82 471 153 147 147
query83 175 151 150 150
query84 271 77 74 74
query85 731 292 316 292
query86 399 307 294 294
query87 4416 4276 4460 4276
query88 3373 2332 2346 2332
query89 392 285 282 282
query90 1964 193 195 193
query91 125 99 98 98
query92 58 53 52 52
query93 1505 544 548 544
query94 709 307 284 284
query95 364 271 264 264
query96 604 273 272 272
query97 3238 3105 3088 3088
query98 234 217 226 217
query99 1625 1286 1282 1282
Total cold run time: 311115 ms
Total hot run time: 195283 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2314387526
<details>
<summary>ClickBench: <b>Total hot run time: 31.9 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cbbeda91329c7f9ecaf9f55ca0b98af79b86d7ee, data reload: false
query1 0.04 0.04 0.04
query2 0.07 0.04 0.04
query3 0.23 0.05 0.05
query4 1.68 0.08 0.09
query5 0.52 0.50 0.50
query6 1.14 0.74 0.73
query7 0.02 0.02 0.01
query8 0.05 0.05 0.05
query9 0.56 0.50 0.49
query10 0.55 0.53 0.54
query11 0.16 0.12 0.12
query12 0.16 0.13 0.13
query13 0.62 0.61 0.59
query14 2.07 2.05 2.14
query15 0.85 0.82 0.83
query16 0.38 0.38 0.38
query17 1.08 1.07 1.01
query18 0.21 0.20 0.21
query19 1.98 1.80 1.68
query20 0.01 0.01 0.01
query21 15.42 0.68 0.66
query22 3.75 7.63 1.74
query23 18.27 1.44 1.28
query24 2.07 0.24 0.22
query25 0.14 0.09 0.08
query26 0.28 0.18 0.17
query27 0.08 0.08 0.07
query28 13.66 1.02 1.00
query29 12.67 3.41 3.34
query30 0.25 0.06 0.05
query31 2.86 0.40 0.39
query32 3.26 0.49 0.48
query33 2.97 3.06 3.02
query34 17.13 4.37 4.42
query35 4.48 4.53 4.43
query36 0.66 0.48 0.48
query37 0.19 0.16 0.15
query38 0.15 0.15 0.16
query39 0.05 0.04 0.04
query40 0.16 0.13 0.14
query41 0.10 0.05 0.05
query42 0.07 0.05 0.06
query43 0.05 0.04 0.04
Total cold run time: 111.1 s
Total hot run time: 31.9 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2272472453
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1706546681
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -788,6 +788,164 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
txn->put(key, val);
}
}
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
Review Comment:
> `partition_versions` is not needed here, since all rowsets belong to the same partition. Read the partition version in here directly.
`convert_tmp_rowsets` is a common api, all roswets not belong to the same partition also work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2290937110
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2286243101
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2286433958
<details>
<summary>TPC-H: <b>Total hot run time: 40288 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 814829fad0b51598e72ff1c3c1e78f4f3225389e, data reload: false
------ Round 1 ----------------------------------
q1 18298 4620 4309 4309
q2 2031 186 182 182
q3 11493 1035 1107 1035
q4 10529 716 700 700
q5 7766 2871 2836 2836
q6 224 140 143 140
q7 969 594 602 594
q8 9352 2072 2112 2072
q9 8621 6549 6576 6549
q10 7070 2253 2201 2201
q11 457 254 256 254
q12 392 222 222 222
q13 18130 2961 2995 2961
q14 270 244 230 230
q15 534 498 498 498
q16 646 510 527 510
q17 992 674 729 674
q18 8147 7359 7419 7359
q19 6085 1089 1023 1023
q20 740 414 411 411
q21 5448 4525 4538 4525
q22 1101 1003 1016 1003
Total cold run time: 119295 ms
Total hot run time: 40288 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4503 4260 4267 4260
q2 393 281 273 273
q3 2944 2779 2758 2758
q4 1945 1758 1743 1743
q5 5639 5621 5540 5540
q6 238 139 138 138
q7 2220 1783 1748 1748
q8 3341 3481 3480 3480
q9 8768 8866 8848 8848
q10 3472 3214 3265 3214
q11 620 528 501 501
q12 848 667 645 645
q13 17066 3144 3151 3144
q14 319 288 287 287
q15 542 498 486 486
q16 635 583 579 579
q17 1821 1524 1549 1524
q18 8226 7975 7871 7871
q19 1868 1605 1708 1605
q20 3147 1963 1993 1963
q21 7727 5299 5379 5299
q22 1129 1043 1052 1043
Total cold run time: 77411 ms
Total hot run time: 56949 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2298075542
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2297065838
<details>
<summary>ClickBench: <b>Total hot run time: 30.99 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26c7330ea2ca69f58795f6cc0b7821111e82c0d5, data reload: false
query1 0.05 0.04 0.04
query2 0.09 0.04 0.05
query3 0.22 0.06 0.04
query4 1.67 0.07 0.07
query5 0.50 0.50 0.50
query6 1.14 0.73 0.72
query7 0.02 0.02 0.02
query8 0.06 0.05 0.05
query9 0.55 0.49 0.50
query10 0.54 0.53 0.56
query11 0.16 0.12 0.12
query12 0.16 0.13 0.12
query13 0.62 0.60 0.59
query14 0.77 0.79 0.77
query15 0.85 0.84 0.82
query16 0.36 0.35 0.39
query17 1.02 1.03 1.04
query18 0.22 0.20 0.19
query19 1.88 1.76 1.73
query20 0.02 0.01 0.01
query21 15.41 0.68 0.65
query22 3.97 8.23 1.79
query23 18.24 1.34 1.34
query24 2.17 0.21 0.22
query25 0.16 0.08 0.09
query26 0.28 0.19 0.17
query27 0.09 0.08 0.08
query28 13.19 1.02 1.00
query29 12.64 3.33 3.30
query30 0.43 0.25 0.24
query31 2.81 0.41 0.41
query32 3.25 0.49 0.50
query33 2.98 2.99 2.97
query34 17.13 4.41 4.37
query35 4.41 4.44 4.47
query36 0.67 0.51 0.51
query37 0.22 0.18 0.17
query38 0.17 0.16 0.16
query39 0.07 0.06 0.05
query40 0.19 0.15 0.15
query41 0.12 0.06 0.06
query42 0.07 0.06 0.06
query43 0.06 0.06 0.06
Total cold run time: 109.63 s
Total hot run time: 30.99 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2245520383
<details>
<summary>TPC-H: <b>Total hot run time: 40400 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7e2b1bd85dd3827c86fb9ead1aa462b016a3aba, data reload: false
------ Round 1 ----------------------------------
q1 18792 7352 4251 4251
q2 2021 202 185 185
q3 10539 1167 1066 1066
q4 10298 778 816 778
q5 7549 2694 2668 2668
q6 222 139 136 136
q7 969 594 583 583
q8 9200 2070 2049 2049
q9 8685 6594 6523 6523
q10 8708 3782 3771 3771
q11 464 232 233 232
q12 387 216 218 216
q13 17801 3008 2997 2997
q14 289 228 241 228
q15 521 476 493 476
q16 498 388 370 370
q17 951 615 671 615
q18 8136 7389 7379 7379
q19 6302 1449 1418 1418
q20 649 329 313 313
q21 4889 3895 3863 3863
q22 339 285 283 283
Total cold run time: 118209 ms
Total hot run time: 40400 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4376 4238 4225 4225
q2 371 275 288 275
q3 3167 2889 2929 2889
q4 2046 1686 1636 1636
q5 5561 5472 5560 5472
q6 227 133 132 132
q7 2198 1765 1833 1765
q8 3272 3391 3371 3371
q9 8706 8901 8628 8628
q10 4037 3781 3873 3781
q11 573 504 492 492
q12 794 622 624 622
q13 17154 3183 3183 3183
q14 319 285 297 285
q15 518 499 479 479
q16 484 440 413 413
q17 1832 1536 1487 1487
q18 8250 7937 7726 7726
q19 1719 1522 1542 1522
q20 2168 1851 1844 1844
q21 5142 4646 4859 4646
q22 561 512 520 512
Total cold run time: 73475 ms
Total hot run time: 55385 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2244658181
<details>
<summary>TPC-DS: <b>Total hot run time: 173045 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0cac9c57dfbabd3aa3c3d90819462ad7e839f881, data reload: false
query1 903 376 374 374
query2 6431 1960 1851 1851
query3 6653 205 210 205
query4 25068 17541 17450 17450
query5 4183 478 478 478
query6 272 162 162 162
query7 4592 280 286 280
query8 237 202 218 202
query9 8631 2398 2375 2375
query10 457 296 272 272
query11 10780 10084 10096 10084
query12 140 90 82 82
query13 1633 371 366 366
query14 10095 7647 7731 7647
query15 226 167 168 167
query16 7766 444 474 444
query17 1587 565 553 553
query18 1827 291 289 289
query19 207 155 156 155
query20 90 84 82 82
query21 212 134 128 128
query22 4411 3962 3969 3962
query23 33979 33094 33140 33094
query24 11103 2927 2807 2807
query25 662 401 390 390
query26 1485 160 154 154
query27 2893 276 273 273
query28 7510 1977 1973 1973
query29 1000 662 629 629
query30 286 156 151 151
query31 976 739 745 739
query32 95 54 65 54
query33 794 351 359 351
query34 900 493 487 487
query35 933 755 762 755
query36 1133 917 918 917
query37 151 84 83 83
query38 2859 2762 2751 2751
query39 868 811 836 811
query40 283 126 120 120
query41 52 46 48 46
query42 113 98 108 98
query43 522 464 474 464
query44 1255 728 722 722
query45 197 165 162 162
query46 1088 822 740 740
query47 1843 1744 1755 1744
query48 351 287 290 287
query49 1138 426 422 422
query50 784 399 390 390
query51 6742 6757 6678 6678
query52 109 94 96 94
query53 366 299 282 282
query54 1020 450 446 446
query55 75 72 71 71
query56 290 283 277 277
query57 1195 1061 1041 1041
query58 256 256 261 256
query59 2924 2817 2677 2677
query60 307 278 298 278
query61 98 117 95 95
query62 828 664 663 663
query63 320 291 301 291
query64 10595 2243 1689 1689
query65 3207 3102 3096 3096
query66 1306 340 340 340
query67 15601 14971 14717 14717
query68 4963 530 543 530
query69 553 337 351 337
query70 1201 1161 1010 1010
query71 459 286 289 286
query72 8821 5691 5593 5593
query73 762 328 323 323
query74 6146 5612 5636 5612
query75 4391 2698 2707 2698
query76 3331 937 893 893
query77 703 320 297 297
query78 11692 9404 9046 9046
query79 7963 520 519 519
query80 1067 488 492 488
query81 596 230 234 230
query82 658 135 130 130
query83 352 167 169 167
query84 275 85 87 85
query85 792 371 301 301
query86 479 296 316 296
query87 3327 3091 3094 3091
query88 4491 2376 2390 2376
query89 510 371 386 371
query90 1961 199 197 197
query91 136 100 104 100
query92 68 51 49 49
query93 4526 503 490 490
query94 1197 295 296 295
query95 409 328 315 315
query96 602 272 268 268
query97 3171 3043 3014 3014
query98 218 204 201 201
query99 1727 1224 1234 1224
Total cold run time: 294440 ms
Total hot run time: 173045 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2250621990
<details>
<summary>TPC-DS: <b>Total hot run time: 173957 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9511fed37daee25d5ec0c6d5a1033b69e5eddf44, data reload: false
query1 907 379 364 364
query2 6450 1920 1771 1771
query3 6645 212 223 212
query4 28599 17564 17378 17378
query5 3757 505 522 505
query6 259 177 163 163
query7 4603 292 294 292
query8 243 199 191 191
query9 8619 2476 2428 2428
query10 424 305 260 260
query11 12272 10097 10053 10053
query12 119 83 88 83
query13 1618 369 367 367
query14 8708 7890 7891 7890
query15 222 170 172 170
query16 7490 490 479 479
query17 1136 560 545 545
query18 1975 286 278 278
query19 200 149 152 149
query20 98 92 89 89
query21 209 102 102 102
query22 4293 4072 4032 4032
query23 34289 34043 33923 33923
query24 11081 2979 2852 2852
query25 622 427 425 425
query26 733 163 160 160
query27 2288 292 283 283
query28 6051 2105 2097 2097
query29 831 471 455 455
query30 272 167 160 160
query31 994 769 760 760
query32 98 54 59 54
query33 719 380 355 355
query34 921 502 497 497
query35 878 756 749 749
query36 1139 999 952 952
query37 140 84 82 82
query38 2990 2857 2820 2820
query39 920 851 853 851
query40 213 122 123 122
query41 47 44 45 44
query42 112 96 96 96
query43 507 467 455 455
query44 1124 726 734 726
query45 204 174 175 174
query46 1088 732 776 732
query47 1830 1769 1754 1754
query48 382 283 308 283
query49 843 417 427 417
query50 802 404 404 404
query51 6707 6663 6709 6663
query52 110 86 94 86
query53 270 190 183 183
query54 858 443 440 440
query55 79 75 72 72
query56 295 287 270 270
query57 1123 1047 1035 1035
query58 242 268 259 259
query59 2864 2593 2607 2593
query60 321 291 281 281
query61 120 93 93 93
query62 787 662 621 621
query63 210 184 187 184
query64 9236 2279 1770 1770
query65 3148 3096 3113 3096
query66 773 330 350 330
query67 15550 14902 14887 14887
query68 4608 553 557 553
query69 539 296 308 296
query70 1102 1138 1123 1123
query71 417 294 328 294
query72 8551 5290 5347 5290
query73 770 374 337 337
query74 6020 5670 5675 5670
query75 3518 2729 2629 2629
query76 2635 953 892 892
query77 658 323 326 323
query78 10463 9003 9161 9003
query79 2349 527 537 527
query80 2598 495 503 495
query81 606 222 216 216
query82 805 137 140 137
query83 324 179 177 177
query84 268 80 83 80
query85 1931 312 308 308
query86 443 308 297 297
query87 3238 3106 3081 3081
query88 4272 2392 2395 2392
query89 399 294 286 286
query90 1937 197 195 195
query91 127 101 100 100
query92 58 52 50 50
query93 2614 555 571 555
query94 769 290 285 285
query95 364 263 339 263
query96 611 268 277 268
query97 3201 3029 3051 3029
query98 228 205 205 205
query99 1740 1255 1267 1255
Total cold run time: 281820 ms
Total hot run time: 173957 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2250578064
<details>
<summary>TPC-H: <b>Total hot run time: 39691 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9511fed37daee25d5ec0c6d5a1033b69e5eddf44, data reload: false
------ Round 1 ----------------------------------
q1 18811 4758 4467 4467
q2 2040 200 196 196
q3 10519 1189 1081 1081
q4 10174 739 745 739
q5 7579 2739 2718 2718
q6 226 140 139 139
q7 966 596 595 595
q8 9232 1919 1946 1919
q9 8840 6535 6598 6535
q10 8827 3787 3812 3787
q11 456 244 247 244
q12 408 225 231 225
q13 19056 2969 2996 2969
q14 283 239 244 239
q15 525 468 501 468
q16 497 393 385 385
q17 980 640 706 640
q18 8179 7546 7485 7485
q19 3813 1068 1077 1068
q20 703 351 339 339
q21 4977 3167 3330 3167
q22 353 287 286 286
Total cold run time: 117444 ms
Total hot run time: 39691 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4467 4261 4262 4261
q2 368 272 266 266
q3 3009 2900 2855 2855
q4 2049 1690 1806 1690
q5 5653 5529 5499 5499
q6 222 137 144 137
q7 2191 1861 1847 1847
q8 3274 3420 3477 3420
q9 8807 8824 8867 8824
q10 4161 3860 3843 3843
q11 612 512 514 512
q12 818 650 644 644
q13 16136 3184 3231 3184
q14 333 298 290 290
q15 533 512 508 508
q16 499 461 446 446
q17 1820 1546 1508 1508
q18 8130 8162 7951 7951
q19 1729 1590 1651 1590
q20 2159 1861 1898 1861
q21 5038 4803 4769 4769
q22 585 485 505 485
Total cold run time: 72593 ms
Total hot run time: 56390 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1708565085
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ break;
Review Comment:
> Even if the tmp_rowset_metas is empty, the TXN might still not be visible. Consider call `make_committed_txn_visible` here.
done, i have removed the `break`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2259590990
<details>
<summary>TPC-DS: <b>Total hot run time: 171014 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8ed44221f04cb0a14f7132c207fbd3cf93e572f1, data reload: false
query1 924 382 383 382
query2 6468 1796 1679 1679
query3 6663 209 223 209
query4 20108 17608 17528 17528
query5 3662 511 519 511
query6 265 168 157 157
query7 4468 299 292 292
query8 261 194 191 191
query9 7635 2391 2388 2388
query10 399 280 268 268
query11 10526 10127 10078 10078
query12 124 90 88 88
query13 1474 400 374 374
query14 9689 8623 8296 8296
query15 204 168 163 163
query16 6823 474 445 445
query17 928 548 548 548
query18 1886 300 278 278
query19 188 147 141 141
query20 91 87 85 85
query21 197 103 98 98
query22 4294 4137 4012 4012
query23 33920 33821 33437 33437
query24 10284 3131 3150 3131
query25 665 408 414 408
query26 1686 164 148 148
query27 2899 273 280 273
query28 7622 2018 1996 1996
query29 1286 431 493 431
query30 248 157 154 154
query31 927 775 775 775
query32 98 62 56 56
query33 652 332 319 319
query34 917 500 497 497
query35 887 773 755 755
query36 1089 923 887 887
query37 296 84 84 84
query38 3011 2853 2805 2805
query39 868 799 804 799
query40 273 114 115 114
query41 46 43 45 43
query42 123 100 96 96
query43 483 448 440 440
query44 1172 747 736 736
query45 207 183 181 181
query46 1086 826 788 788
query47 1785 1700 1703 1700
query48 367 300 304 300
query49 959 455 414 414
query50 923 432 437 432
query51 6761 6808 6629 6629
query52 95 94 87 87
query53 252 184 185 184
query54 582 446 459 446
query55 78 74 76 74
query56 266 253 248 248
query57 1131 1032 1000 1000
query58 282 263 266 263
query59 2793 2398 2502 2398
query60 291 279 270 270
query61 94 94 96 94
query62 875 659 655 655
query63 218 183 190 183
query64 5666 1944 1874 1874
query65 3190 3080 3095 3080
query66 1321 337 325 325
query67 15311 14794 14663 14663
query68 4227 547 564 547
query69 543 346 312 312
query70 1105 1029 1118 1029
query71 412 286 277 277
query72 7008 2714 2474 2474
query73 765 332 329 329
query74 5984 5697 5561 5561
query75 3406 2732 2744 2732
query76 2318 1347 1412 1347
query77 406 313 310 310
query78 9328 8954 8944 8944
query79 1495 541 535 535
query80 938 498 547 498
query81 556 226 227 226
query82 823 127 124 124
query83 258 170 173 170
query84 265 79 79 79
query85 1412 305 296 296
query86 491 304 319 304
query87 3356 3077 3108 3077
query88 3702 2397 2409 2397
query89 380 288 295 288
query90 1779 188 194 188
query91 124 99 101 99
query92 63 51 51 51
query93 2069 613 617 613
query94 776 309 285 285
query95 384 266 266 266
query96 612 283 275 275
query97 3214 3017 3046 3017
query98 230 206 197 197
query99 1678 1308 1284 1284
Total cold run time: 261369 ms
Total hot run time: 171014 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2274915525
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2245374489
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2245549983
<details>
<summary>TPC-DS: <b>Total hot run time: 174268 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7e2b1bd85dd3827c86fb9ead1aa462b016a3aba, data reload: false
query1 912 373 376 373
query2 6371 1871 1800 1800
query3 6641 204 216 204
query4 28243 17484 17186 17186
query5 3760 466 462 462
query6 262 188 172 172
query7 4573 294 281 281
query8 238 192 189 189
query9 8532 2427 2431 2427
query10 441 295 275 275
query11 11762 10107 10087 10087
query12 114 81 81 81
query13 1623 369 353 353
query14 10048 7513 7638 7513
query15 215 162 158 158
query16 7488 469 457 457
query17 1384 573 513 513
query18 1847 275 269 269
query19 195 145 147 145
query20 88 80 79 79
query21 202 127 123 123
query22 4267 4026 3993 3993
query23 34119 33647 33662 33647
query24 10994 2936 2928 2928
query25 620 386 396 386
query26 708 155 149 149
query27 2423 273 278 273
query28 6388 2089 2106 2089
query29 911 627 637 627
query30 278 154 155 154
query31 975 778 769 769
query32 101 54 53 53
query33 720 334 335 334
query34 904 495 493 493
query35 877 758 756 756
query36 1166 962 973 962
query37 141 83 82 82
query38 2939 2882 2979 2882
query39 865 833 800 800
query40 203 121 119 119
query41 46 44 47 44
query42 113 96 102 96
query43 500 459 460 459
query44 1137 727 728 727
query45 192 162 162 162
query46 1101 761 732 732
query47 1849 1796 1756 1756
query48 371 295 300 295
query49 868 416 425 416
query50 765 385 383 383
query51 6757 6699 6656 6656
query52 103 91 91 91
query53 361 293 283 283
query54 955 454 456 454
query55 75 73 72 72
query56 301 280 288 280
query57 1137 1033 1060 1033
query58 268 282 292 282
query59 2739 2670 2450 2450
query60 314 291 288 288
query61 120 115 120 115
query62 803 662 646 646
query63 318 289 288 288
query64 9235 2287 7542 2287
query65 3127 3116 3090 3090
query66 760 340 346 340
query67 15665 15032 14933 14933
query68 6166 547 555 547
query69 768 466 383 383
query70 1316 1158 1095 1095
query71 501 274 285 274
query72 8876 5462 5718 5462
query73 752 327 324 324
query74 6289 5716 5739 5716
query75 4470 2712 2672 2672
query76 4033 922 930 922
query77 732 299 294 294
query78 9595 8950 8940 8940
query79 7861 522 523 522
query80 2347 489 472 472
query81 590 223 260 223
query82 1092 140 137 137
query83 282 164 163 163
query84 278 84 84 84
query85 1329 317 306 306
query86 448 304 321 304
query87 3272 3077 3103 3077
query88 5184 2406 2412 2406
query89 510 393 385 385
query90 1984 199 192 192
query91 129 98 104 98
query92 61 49 48 48
query93 6449 519 527 519
query94 1239 282 291 282
query95 413 321 332 321
query96 629 281 272 272
query97 3187 3042 3046 3042
query98 211 210 241 210
query99 1589 1286 1275 1275
Total cold run time: 297100 ms
Total hot run time: 174268 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2245114473
<details>
<summary>ClickBench: <b>Total hot run time: 31.44 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d47f33a15e68079183da96a15302b529dc1c36a2, data reload: false
query1 0.05 0.04 0.04
query2 0.07 0.04 0.04
query3 0.23 0.05 0.05
query4 1.67 0.08 0.08
query5 0.52 0.49 0.50
query6 1.12 0.73 0.74
query7 0.02 0.01 0.01
query8 0.06 0.05 0.04
query9 0.55 0.50 0.49
query10 0.55 0.55 0.54
query11 0.15 0.12 0.12
query12 0.15 0.12 0.12
query13 0.59 0.59 0.58
query14 0.76 0.77 0.78
query15 0.85 0.82 0.81
query16 0.35 0.35 0.38
query17 0.96 0.94 0.96
query18 0.23 0.22 0.22
query19 1.77 1.68 1.73
query20 0.01 0.01 0.01
query21 15.39 0.74 0.65
query22 4.62 6.41 2.77
query23 18.36 1.42 1.28
query24 2.11 0.22 0.23
query25 0.14 0.09 0.09
query26 0.29 0.21 0.21
query27 0.45 0.24 0.24
query28 13.27 1.02 1.02
query29 12.62 3.30 3.29
query30 0.25 0.06 0.05
query31 2.90 0.40 0.40
query32 3.23 0.49 0.47
query33 2.87 2.89 2.95
query34 17.08 4.33 4.37
query35 4.41 4.39 4.40
query36 0.65 0.45 0.46
query37 0.19 0.16 0.15
query38 0.16 0.15 0.14
query39 0.05 0.04 0.04
query40 0.15 0.12 0.12
query41 0.09 0.04 0.05
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 110.04 s
Total hot run time: 31.44 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1733771936
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1119,6 +1200,12 @@ int InstanceRecycler::recycle_tablets(int64_t table_id, int64_t index_id, int64_
return -1;
}
int64_t tablet_id = tablet_meta_pb.tablet_id();
+
+ if (!check_lazy_txn_finished(txn_kv_, instance_id_, tablet_meta_pb.tablet_id())) {
+ LOG(WARNING) << "lazy txn not finished tablet_id=" << tablet_meta_pb.tablet_id();
Review Comment:
> LOG "lazy txn not finished, skip recycling. tablet_id= ... txn_id ..." and add a bvar to monitor `skip_recyle_tablet_lazy_txn`
int the `check_lazy_txn_finished `, every abnormal branch has more detail log
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1119,6 +1200,12 @@ int InstanceRecycler::recycle_tablets(int64_t table_id, int64_t index_id, int64_
return -1;
}
int64_t tablet_id = tablet_meta_pb.tablet_id();
+
+ if (!check_lazy_txn_finished(txn_kv_, instance_id_, tablet_meta_pb.tablet_id())) {
+ LOG(WARNING) << "lazy txn not finished tablet_id=" << tablet_meta_pb.tablet_id();
Review Comment:
> LOG "lazy txn not finished, skip recycling. tablet_id= ... txn_id ..." and add a bvar to monitor `skip_recyle_tablet_lazy_txn`
in the `check_lazy_txn_finished `, every abnormal branch has more detail log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1733036625
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,475 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
Review Comment:
warning: function 'convert_tmp_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void convert_tmp_rowsets(
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:40:** 154 lines including whitespace and comments (threshold 80)
```cpp
void convert_tmp_rowsets(
^
```
</details>
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,475 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+ DCHECK(version_pb.txn_ids_size() > 0);
+ DCHECK_EQ(version_pb.txn_ids(0), txn_id);
+
+ int64_t version = version_pb.has_version() ? (version_pb.version() + 1) : 2;
+
+ std::string rowset_key = meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ // tmp rowset key has been converted
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version);
+ tmp_rowset_pb.set_end_version(version);
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
Review Comment:
warning: function 'make_committed_txn_visible' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:200:** 82 lines including whitespace and comments (threshold 80)
```cpp
void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
^
```
</details>
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,475 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+ DCHECK(version_pb.txn_ids_size() > 0);
+ DCHECK_EQ(version_pb.txn_ids(0), txn_id);
+
+ int64_t version = version_pb.has_version() ? (version_pb.version() + 1) : 2;
+
+ std::string rowset_key = meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ // tmp rowset key has been converted
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version);
+ tmp_rowset_pb.set_end_version(version);
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(recycle_key, recycle_val);
+ LOG(INFO) << "put recycle_key=" << hex(recycle_key) << " txn_id=" << txn_id;
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:296:** 138 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312931990
<details>
<summary>TPC-H: <b>Total hot run time: 38230 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40557e2131ead7df254d700743017f557f0777e6, data reload: false
------ Round 1 ----------------------------------
q1 17711 4543 4324 4324
q2 2018 199 171 171
q3 11601 990 1079 990
q4 10486 659 727 659
q5 7748 2839 2779 2779
q6 225 137 137 137
q7 985 617 612 612
q8 9318 2062 2084 2062
q9 7205 6534 6530 6530
q10 7012 2266 2255 2255
q11 449 246 251 246
q12 401 219 224 219
q13 17765 3017 3012 3012
q14 277 228 225 225
q15 532 483 488 483
q16 489 398 389 389
q17 974 636 726 636
q18 7507 6936 6959 6936
q19 1398 1086 1052 1052
q20 699 324 339 324
q21 4320 3156 3167 3156
q22 1120 1033 1033 1033
Total cold run time: 110240 ms
Total hot run time: 38230 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4368 4350 4321 4321
q2 382 276 269 269
q3 2895 2682 2687 2682
q4 1944 1589 1654 1589
q5 5615 5696 5751 5696
q6 231 139 142 139
q7 2246 1859 1790 1790
q8 3292 3435 3491 3435
q9 8903 8729 8782 8729
q10 3568 3422 3394 3394
q11 607 533 514 514
q12 830 674 636 636
q13 13700 3194 3203 3194
q14 323 291 291 291
q15 530 501 503 501
q16 513 448 435 435
q17 1844 1527 1533 1527
q18 8147 7748 7744 7744
q19 1752 1507 1601 1507
q20 2145 1933 1929 1929
q21 5787 5364 5580 5364
q22 1099 1059 1058 1058
Total cold run time: 70721 ms
Total hot run time: 56744 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2314139153
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2317508011
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732239346
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
Review Comment:
> log
Add a log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1713235462
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -1405,104 +1438,144 @@ void MetaServiceImpl::get_rowset(::google::protobuf::RpcController* controller,
int64_t req_cc_cnt = request->cumulative_compaction_cnt();
int64_t req_cp = request->cumulative_point();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv_->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- msg = "failed to create txn";
- return;
- }
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ msg = "failed to create txn";
+ return;
+ }
- TabletIndexPB idx(request->idx());
- // Get tablet id index from kv
- if (!idx.has_table_id() || !idx.has_index_id() || !idx.has_partition_id()) {
+ TabletIndexPB idx;
+ // Get tablet id index from kv
get_tablet_idx(code, msg, txn.get(), instance_id, tablet_id, idx);
- if (code != MetaServiceCode::OK) return;
- }
- // TODO(plat1ko): Judge if tablet has been dropped (in dropped index/partition)
-
- TabletStatsPB tablet_stat;
- internal_get_tablet_stats(code, msg, txn.get(), instance_id, idx, tablet_stat, true);
- if (code != MetaServiceCode::OK) return;
- VLOG_DEBUG << "tablet_id=" << tablet_id << " stats=" << proto_to_json(tablet_stat);
-
- int64_t bc_cnt = tablet_stat.base_compaction_cnt();
- int64_t cc_cnt = tablet_stat.cumulative_compaction_cnt();
- int64_t cp = tablet_stat.cumulative_point();
-
- response->mutable_stats()->CopyFrom(tablet_stat);
-
- int64_t req_start = request->start_version();
- int64_t req_end = request->end_version();
- req_end = req_end < 0 ? std::numeric_limits<int64_t>::max() - 1 : req_end;
-
- //==========================================================================
- // Find version ranges to be synchronized due to compaction
- //==========================================================================
- if (req_bc_cnt > bc_cnt || req_cc_cnt > cc_cnt || req_cp > cp) {
- code = MetaServiceCode::INVALID_ARGUMENT;
- ss << "no valid compaction_cnt or cumulative_point given. req_bc_cnt=" << req_bc_cnt
- << ", bc_cnt=" << bc_cnt << ", req_cc_cnt=" << req_cc_cnt << ", cc_cnt=" << cc_cnt
- << ", req_cp=" << req_cp << ", cp=" << cp;
- msg = ss.str();
- return;
- }
- auto versions = calc_sync_versions(req_bc_cnt, bc_cnt, req_cc_cnt, cc_cnt, req_cp, cp,
- req_start, req_end);
- for (auto [start, end] : versions) {
- internal_get_rowset(txn.get(), start, end, instance_id, tablet_id, code, msg, response);
if (code != MetaServiceCode::OK) {
return;
}
- }
+ DCHECK(request->has_idx());
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id, idx.db_id(), idx.table_id(), idx.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, tablet_id=" << tablet_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // get referenced schema
- std::unordered_map<int32_t, doris::TabletSchemaCloudPB*> version_to_schema;
- for (auto& rowset_meta : *response->mutable_rowset_meta()) {
- if (rowset_meta.has_tablet_schema()) {
- version_to_schema.emplace(rowset_meta.tablet_schema().schema_version(),
- rowset_meta.mutable_tablet_schema());
- rowset_meta.set_schema_version(rowset_meta.tablet_schema().schema_version());
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb tablet_id=" << tablet_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
}
- rowset_meta.set_index_id(idx.index_id());
- }
- bool need_read_schema_dict = false;
- auto arena = response->GetArena();
- for (auto& rowset_meta : *response->mutable_rowset_meta()) {
- if (rowset_meta.has_schema_dict_key_list()) {
- need_read_schema_dict = true;
+
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
+
+ std::tie(code, msg) = task->wait();
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << version_pb.txn_id()
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
+ continue;
}
- if (rowset_meta.has_tablet_schema()) continue;
- if (!rowset_meta.has_schema_version()) {
+
+ // TODO(plat1ko): Judge if tablet has been dropped (in dropped index/partition)
+
+ TabletStatsPB tablet_stat;
+ internal_get_tablet_stats(code, msg, txn.get(), instance_id, idx, tablet_stat, true);
+ if (code != MetaServiceCode::OK) return;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (code != MetaServiceCode::OK) { return;
}
```
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -1405,104 +1438,144 @@
int64_t req_cc_cnt = request->cumulative_compaction_cnt();
int64_t req_cp = request->cumulative_point();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv_->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- msg = "failed to create txn";
- return;
- }
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ msg = "failed to create txn";
+ return;
+ }
- TabletIndexPB idx(request->idx());
- // Get tablet id index from kv
- if (!idx.has_table_id() || !idx.has_index_id() || !idx.has_partition_id()) {
+ TabletIndexPB idx;
+ // Get tablet id index from kv
get_tablet_idx(code, msg, txn.get(), instance_id, tablet_id, idx);
- if (code != MetaServiceCode::OK) return;
- }
- // TODO(plat1ko): Judge if tablet has been dropped (in dropped index/partition)
-
- TabletStatsPB tablet_stat;
- internal_get_tablet_stats(code, msg, txn.get(), instance_id, idx, tablet_stat, true);
- if (code != MetaServiceCode::OK) return;
- VLOG_DEBUG << "tablet_id=" << tablet_id << " stats=" << proto_to_json(tablet_stat);
-
- int64_t bc_cnt = tablet_stat.base_compaction_cnt();
- int64_t cc_cnt = tablet_stat.cumulative_compaction_cnt();
- int64_t cp = tablet_stat.cumulative_point();
-
- response->mutable_stats()->CopyFrom(tablet_stat);
-
- int64_t req_start = request->start_version();
- int64_t req_end = request->end_version();
- req_end = req_end < 0 ? std::numeric_limits<int64_t>::max() - 1 : req_end;
-
- //==========================================================================
- // Find version ranges to be synchronized due to compaction
- //==========================================================================
- if (req_bc_cnt > bc_cnt || req_cc_cnt > cc_cnt || req_cp > cp) {
- code = MetaServiceCode::INVALID_ARGUMENT;
- ss << "no valid compaction_cnt or cumulative_point given. req_bc_cnt=" << req_bc_cnt
- << ", bc_cnt=" << bc_cnt << ", req_cc_cnt=" << req_cc_cnt << ", cc_cnt=" << cc_cnt
- << ", req_cp=" << req_cp << ", cp=" << cp;
- msg = ss.str();
- return;
- }
- auto versions = calc_sync_versions(req_bc_cnt, bc_cnt, req_cc_cnt, cc_cnt, req_cp, cp,
- req_start, req_end);
- for (auto [start, end] : versions) {
- internal_get_rowset(txn.get(), start, end, instance_id, tablet_id, code, msg, response);
if (code != MetaServiceCode::OK) {
return;
}
- }
+ DCHECK(request->has_idx());
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id, idx.db_id(), idx.table_id(), idx.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, tablet_id=" << tablet_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // get referenced schema
- std::unordered_map<int32_t, doris::TabletSchemaCloudPB*> version_to_schema;
- for (auto& rowset_meta : *response->mutable_rowset_meta()) {
- if (rowset_meta.has_tablet_schema()) {
- version_to_schema.emplace(rowset_meta.tablet_schema().schema_version(),
- rowset_meta.mutable_tablet_schema());
- rowset_meta.set_schema_version(rowset_meta.tablet_schema().schema_version());
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb tablet_id=" << tablet_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
}
- rowset_meta.set_index_id(idx.index_id());
- }
- bool need_read_schema_dict = false;
- auto arena = response->GetArena();
- for (auto& rowset_meta : *response->mutable_rowset_meta()) {
- if (rowset_meta.has_schema_dict_key_list()) {
- need_read_schema_dict = true;
+
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
+
+ std::tie(code, msg) = task->wait();
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << version_pb.txn_id()
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
+ continue;
}
- if (rowset_meta.has_tablet_schema()) continue;
- if (!rowset_meta.has_schema_version()) {
+
+ // TODO(plat1ko): Judge if tablet has been dropped (in dropped index/partition)
+
+ TabletStatsPB tablet_stat;
+ internal_get_tablet_stats(code, msg, txn.get(), instance_id, idx, tablet_stat, true);
+ if (code != MetaServiceCode::OK) return;
+ VLOG_DEBUG << "tablet_id=" << tablet_id << " stats=" << proto_to_json(tablet_stat);
+
+ int64_t bc_cnt = tablet_stat.base_compaction_cnt();
+ int64_t cc_cnt = tablet_stat.cumulative_compaction_cnt();
+ int64_t cp = tablet_stat.cumulative_point();
+
+ response->mutable_stats()->CopyFrom(tablet_stat);
+
+ int64_t req_start = request->start_version();
+ int64_t req_end = request->end_version();
+ req_end = req_end < 0 ? std::numeric_limits<int64_t>::max() - 1 : req_end;
+
+ //==========================================================================
+ // Find version ranges to be synchronized due to compaction
+ //==========================================================================
+ if (req_bc_cnt > bc_cnt || req_cc_cnt > cc_cnt || req_cp > cp) {
code = MetaServiceCode::INVALID_ARGUMENT;
- msg = fmt::format(
- "rowset_meta must have either schema or schema_version, "
- "rowset_version=[{}-{}]",
- rowset_meta.start_version(), rowset_meta.end_version());
+ ss << "no valid compaction_cnt or cumulative_point given. req_bc_cnt=" << req_bc_cnt
+ << ", bc_cnt=" << bc_cnt << ", req_cc_cnt=" << req_cc_cnt << ", cc_cnt=" << cc_cnt
+ << ", req_cp=" << req_cp << ", cp=" << cp;
+ msg = ss.str();
return;
}
- if (auto it = version_to_schema.find(rowset_meta.schema_version());
- it != version_to_schema.end()) {
- if (arena != nullptr) {
- rowset_meta.set_allocated_tablet_schema(it->second);
- } else {
- rowset_meta.mutable_tablet_schema()->CopyFrom(*it->second);
+ auto versions = calc_sync_versions(req_bc_cnt, bc_cnt, req_cc_cnt, cc_cnt, req_cp, cp,
+ req_start, req_end);
+ for (auto [start, end] : versions) {
+ internal_get_rowset(txn.get(), start, end, instance_id, tablet_id, code, msg, response);
+ if (code != MetaServiceCode::OK) {
+ return;
}
- } else {
- auto key = meta_schema_key({instance_id, idx.index_id(), rowset_meta.schema_version()});
- if (!try_fetch_and_parse_schema(txn.get(), rowset_meta, key, code, msg)) {
+ }
+
+ // get referenced schema
+ std::unordered_map<int32_t, doris::TabletSchemaCloudPB*> version_to_schema;
+ for (auto& rowset_meta : *response->mutable_rowset_meta()) {
+ if (rowset_meta.has_tablet_schema()) {
+ version_to_schema.emplace(rowset_meta.tablet_schema().schema_version(),
+ rowset_meta.mutable_tablet_schema());
+ rowset_meta.set_schema_version(rowset_meta.tablet_schema().schema_version());
+ }
+ rowset_meta.set_index_id(idx.index_id());
+ }
+ bool need_read_schema_dict = false;
+ auto arena = response->GetArena();
+ for (auto& rowset_meta : *response->mutable_rowset_meta()) {
+ if (rowset_meta.has_schema_dict_key_list()) {
+ need_read_schema_dict = true;
+ }
+ if (rowset_meta.has_tablet_schema()) continue;
+ if (!rowset_meta.has_schema_version()) {
+ code = MetaServiceCode::INVALID_ARGUMENT;
+ msg = fmt::format(
+ "rowset_meta must have either schema or schema_version, "
+ "rowset_version=[{}-{}]",
+ rowset_meta.start_version(), rowset_meta.end_version());
return;
}
- version_to_schema.emplace(rowset_meta.schema_version(),
- rowset_meta.mutable_tablet_schema());
+ if (auto it = version_to_schema.find(rowset_meta.schema_version());
+ it != version_to_schema.end()) {
+ if (arena != nullptr) {
+ rowset_meta.set_allocated_tablet_schema(it->second);
+ } else {
+ rowset_meta.mutable_tablet_schema()->CopyFrom(*it->second);
+ }
+ } else {
+ auto key = meta_schema_key({instance_id, idx.index_id(), rowset_meta.schema_version()});
+ if (!try_fetch_and_parse_schema(txn.get(), rowset_meta, key, code, msg)) {
+ return;
+ }
+ version_to_schema.emplace(rowset_meta.schema_version(),
+ rowset_meta.mutable_tablet_schema());
+ }
}
- }
- if (need_read_schema_dict) {
- read_schema_from_dict(code, msg, instance_id, idx.index_id(), txn.get(),
- response->mutable_rowset_meta());
- if (code != MetaServiceCode::OK) return;
- }
+ if (need_read_schema_dict) {
+ read_schema_from_dict(code, msg, instance_id, idx.index_id(), txn.get(),
+ response->mutable_rowset_meta());
+ if (code != MetaServiceCode::OK) return;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (code != MetaServiceCode::OK) { return;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1691448015
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+extern void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+extern void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+extern void make_committed_txn_visible(const std::string& instance_id, int64_t db_id,
+ int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id,
Review Comment:
warning: function 'TxnLazyCommitTask' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id,
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:41:** 130 lines including whitespace and comments (threshold 80)
```cpp
TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id,
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2249289401
<details>
<summary>ClickBench: <b>Total hot run time: 30.6 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7e2b1bd85dd3827c86fb9ead1aa462b016a3aba, data reload: false
query1 0.04 0.04 0.03
query2 0.10 0.04 0.04
query3 0.22 0.07 0.06
query4 1.66 0.10 0.10
query5 0.50 0.49 0.47
query6 1.13 0.73 0.73
query7 0.02 0.01 0.02
query8 0.04 0.04 0.04
query9 0.54 0.49 0.50
query10 0.54 0.55 0.54
query11 0.15 0.11 0.12
query12 0.15 0.13 0.13
query13 0.59 0.59 0.58
query14 0.78 0.77 0.77
query15 0.86 0.81 0.82
query16 0.37 0.35 0.36
query17 1.04 0.96 1.00
query18 0.23 0.22 0.21
query19 1.76 1.67 1.76
query20 0.01 0.01 0.01
query21 15.39 0.78 0.65
query22 4.40 7.39 1.91
query23 18.28 1.43 1.28
query24 2.08 0.25 0.22
query25 0.16 0.08 0.08
query26 0.31 0.22 0.22
query27 0.45 0.23 0.23
query28 13.27 1.02 1.01
query29 12.57 3.31 3.27
query30 0.26 0.06 0.06
query31 2.88 0.40 0.38
query32 3.27 0.48 0.49
query33 2.90 2.91 2.92
query34 16.97 4.33 4.33
query35 4.42 4.40 4.42
query36 0.67 0.49 0.48
query37 0.19 0.16 0.16
query38 0.16 0.14 0.14
query39 0.05 0.03 0.03
query40 0.15 0.13 0.12
query41 0.09 0.04 0.05
query42 0.05 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.74 s
Total hot run time: 30.6 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2260719555
<details>
<summary>ClickBench: <b>Total hot run time: 30.38 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f23de131c772e5e5b480379dddd6cae001bf04a6, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.04 0.05
query3 0.23 0.05 0.04
query4 1.67 0.07 0.07
query5 0.48 0.48 0.48
query6 1.15 0.72 0.72
query7 0.02 0.01 0.01
query8 0.05 0.04 0.05
query9 0.58 0.52 0.52
query10 0.60 0.56 0.55
query11 0.15 0.11 0.11
query12 0.15 0.13 0.13
query13 0.62 0.61 0.60
query14 0.77 0.80 0.80
query15 0.89 0.87 0.87
query16 0.35 0.35 0.36
query17 0.97 1.00 1.01
query18 0.22 0.22 0.22
query19 1.87 1.77 1.73
query20 0.02 0.01 0.01
query21 15.38 0.80 0.65
query22 3.83 7.61 1.38
query23 18.02 1.42 1.40
query24 2.28 0.22 0.21
query25 0.18 0.09 0.09
query26 0.33 0.22 0.22
query27 0.45 0.24 0.24
query28 13.15 1.01 0.98
query29 12.59 3.35 3.36
query30 0.26 0.06 0.07
query31 2.84 0.41 0.41
query32 3.24 0.51 0.48
query33 2.95 2.95 2.99
query34 15.43 4.27 4.26
query35 4.31 4.28 4.30
query36 0.68 0.49 0.50
query37 0.19 0.17 0.17
query38 0.16 0.15 0.15
query39 0.04 0.03 0.04
query40 0.16 0.13 0.13
query41 0.10 0.06 0.05
query42 0.05 0.04 0.04
query43 0.05 0.04 0.04
Total cold run time: 107.57 s
Total hot run time: 30.38 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2261901251
<details>
<summary>TPC-DS: <b>Total hot run time: 169623 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c0a6c5ac10a4aa5587bfabcd3b74e29aa97bef62, data reload: false
query1 916 387 373 373
query2 6441 1716 1653 1653
query3 6654 213 229 213
query4 19174 17504 17339 17339
query5 3627 530 510 510
query6 288 165 174 165
query7 4627 314 296 296
query8 248 193 183 183
query9 8515 2432 2426 2426
query10 415 263 258 258
query11 10738 9918 10013 9918
query12 117 86 90 86
query13 1644 385 363 363
query14 9036 7071 7760 7071
query15 209 158 170 158
query16 6944 444 451 444
query17 919 552 539 539
query18 1916 284 288 284
query19 189 138 139 138
query20 90 88 81 81
query21 199 100 97 97
query22 4285 4057 4105 4057
query23 33745 33678 33404 33404
query24 10363 3176 3128 3128
query25 696 408 408 408
query26 2031 158 153 153
query27 3130 284 285 284
query28 7564 2058 2035 2035
query29 1346 437 452 437
query30 230 153 158 153
query31 967 759 777 759
query32 101 59 57 57
query33 669 315 345 315
query34 950 505 528 505
query35 894 790 760 760
query36 1024 855 862 855
query37 225 85 81 81
query38 2924 2800 2776 2776
query39 875 812 819 812
query40 261 115 112 112
query41 47 43 45 43
query42 126 100 97 97
query43 457 397 427 397
query44 1201 744 732 732
query45 205 176 178 176
query46 1083 813 788 788
query47 1824 1737 1723 1723
query48 366 298 301 298
query49 940 430 419 419
query50 909 436 441 436
query51 6654 6729 6688 6688
query52 103 87 89 87
query53 249 181 186 181
query54 662 457 462 457
query55 76 73 74 73
query56 288 249 246 246
query57 1162 1026 1053 1026
query58 278 272 303 272
query59 2626 2449 2407 2407
query60 294 279 261 261
query61 94 93 93 93
query62 871 673 656 656
query63 220 183 189 183
query64 5699 1910 1892 1892
query65 3179 3086 3107 3086
query66 1241 347 328 328
query67 15076 14712 14793 14712
query68 4321 572 571 571
query69 446 317 301 301
query70 1113 976 1067 976
query71 369 279 274 274
query72 7581 2752 2480 2480
query73 763 332 336 332
query74 6086 5643 5553 5553
query75 3404 2740 2730 2730
query76 2272 1342 1419 1342
query77 420 305 308 305
query78 9381 8930 8986 8930
query79 1878 545 528 528
query80 1109 501 557 501
query81 554 225 219 219
query82 1049 129 126 126
query83 237 175 167 167
query84 281 79 77 77
query85 1300 310 302 302
query86 401 298 297 297
query87 3279 3116 3084 3084
query88 3095 2513 2497 2497
query89 389 287 286 286
query90 1683 197 195 195
query91 128 101 96 96
query92 56 50 47 47
query93 1475 625 629 625
query94 824 295 300 295
query95 380 265 281 265
query96 593 279 285 279
query97 3226 3035 3030 3030
query98 218 198 191 191
query99 1662 1279 1287 1279
Total cold run time: 261089 ms
Total hot run time: 169623 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2262331605
<details>
<summary>ClickBench: <b>Total hot run time: 30.72 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f40d3af909e3a4ce446ee5217827a6811bce34d1, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.04
query3 0.23 0.05 0.04
query4 1.71 0.07 0.09
query5 0.49 0.49 0.50
query6 1.14 0.72 0.72
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.56 0.52 0.51
query10 0.57 0.57 0.55
query11 0.16 0.12 0.12
query12 0.15 0.12 0.12
query13 0.61 0.60 0.60
query14 0.78 0.79 0.77
query15 0.90 0.86 0.87
query16 0.35 0.36 0.35
query17 1.01 0.98 0.98
query18 0.22 0.20 0.20
query19 1.87 1.75 1.73
query20 0.01 0.01 0.01
query21 15.40 0.78 0.67
query22 3.84 6.74 2.14
query23 18.03 1.28 1.21
query24 2.24 0.23 0.23
query25 0.18 0.08 0.07
query26 0.32 0.21 0.21
query27 0.46 0.24 0.23
query28 13.16 1.01 0.97
query29 12.53 3.32 3.29
query30 0.26 0.05 0.06
query31 2.85 0.41 0.40
query32 3.24 0.49 0.50
query33 2.95 2.94 3.00
query34 15.44 4.26 4.29
query35 4.29 4.30 4.26
query36 0.67 0.47 0.48
query37 0.19 0.17 0.16
query38 0.16 0.14 0.15
query39 0.04 0.04 0.03
query40 0.16 0.13 0.13
query41 0.10 0.05 0.05
query42 0.06 0.04 0.05
query43 0.04 0.05 0.04
Total cold run time: 107.55 s
Total hot run time: 30.72 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2261914348
<details>
<summary>ClickBench: <b>Total hot run time: 30.23 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c0a6c5ac10a4aa5587bfabcd3b74e29aa97bef62, data reload: false
query1 0.04 0.04 0.03
query2 0.07 0.03 0.04
query3 0.23 0.04 0.05
query4 1.69 0.08 0.08
query5 0.48 0.48 0.48
query6 1.14 0.72 0.71
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.56 0.51 0.50
query10 0.59 0.56 0.57
query11 0.16 0.11 0.12
query12 0.15 0.13 0.13
query13 0.62 0.60 0.60
query14 0.78 0.78 0.81
query15 0.91 0.86 0.86
query16 0.36 0.35 0.35
query17 0.98 0.99 0.99
query18 0.22 0.21 0.21
query19 1.83 1.72 1.71
query20 0.01 0.01 0.01
query21 15.40 0.80 0.67
query22 4.36 7.85 1.52
query23 18.10 1.28 1.30
query24 2.26 0.23 0.22
query25 0.18 0.09 0.08
query26 0.32 0.21 0.21
query27 0.46 0.23 0.23
query28 13.16 1.00 0.97
query29 12.53 3.33 3.32
query30 0.26 0.06 0.06
query31 2.87 0.43 0.41
query32 3.21 0.49 0.50
query33 2.95 2.96 2.93
query34 15.44 4.23 4.26
query35 4.30 4.31 4.29
query36 0.69 0.47 0.49
query37 0.18 0.17 0.16
query38 0.18 0.15 0.15
query39 0.04 0.04 0.03
query40 0.16 0.14 0.14
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 108.15 s
Total hot run time: 30.23 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2302361848
<details>
<summary>ClickBench: <b>Total hot run time: 31.6 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3ecb308aa0ea6ae75918b441fcce5ed28f71c479, data reload: false
query1 0.05 0.04 0.04
query2 0.08 0.04 0.04
query3 0.23 0.06 0.06
query4 1.66 0.07 0.08
query5 0.54 0.52 0.50
query6 1.13 0.72 0.73
query7 0.03 0.02 0.02
query8 0.05 0.05 0.05
query9 0.55 0.48 0.49
query10 0.55 0.55 0.56
query11 0.17 0.13 0.12
query12 0.17 0.13 0.12
query13 0.62 0.59 0.58
query14 0.77 0.81 0.79
query15 0.84 0.82 0.83
query16 0.38 0.37 0.38
query17 1.07 1.06 1.04
query18 0.23 0.21 0.21
query19 1.98 1.87 1.83
query20 0.02 0.01 0.01
query21 15.40 0.68 0.67
query22 3.98 7.35 2.12
query23 18.25 1.44 1.32
query24 2.12 0.23 0.23
query25 0.15 0.08 0.08
query26 0.26 0.19 0.18
query27 0.08 0.08 0.08
query28 13.22 1.03 1.02
query29 12.66 3.37 3.36
query30 0.42 0.24 0.24
query31 2.80 0.42 0.41
query32 3.24 0.49 0.48
query33 2.99 2.99 2.98
query34 16.99 4.40 4.36
query35 4.42 4.42 4.45
query36 0.68 0.50 0.49
query37 0.21 0.18 0.17
query38 0.17 0.18 0.17
query39 0.07 0.06 0.05
query40 0.19 0.16 0.16
query41 0.12 0.06 0.07
query42 0.08 0.07 0.07
query43 0.08 0.06 0.06
Total cold run time: 109.7 s
Total hot run time: 31.6 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2298372481
<details>
<summary>TPC-H: <b>Total hot run time: 38269 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1d63b327147196a6029b3421305f1eed1a00dc18, data reload: false
------ Round 1 ----------------------------------
q1 18209 4465 4317 4317
q2 2056 216 226 216
q3 11523 945 1169 945
q4 10537 775 665 665
q5 7805 2866 2785 2785
q6 267 161 158 158
q7 1015 660 660 660
q8 9397 2126 2107 2107
q9 7276 6613 6573 6573
q10 7102 2249 2271 2249
q11 492 271 270 270
q12 427 263 262 262
q13 18121 3055 3032 3032
q14 308 251 251 251
q15 564 533 554 533
q16 540 428 418 418
q17 988 646 753 646
q18 7562 6797 6824 6797
q19 6509 1076 1038 1038
q20 727 356 379 356
q21 4291 2948 2982 2948
q22 1172 1067 1043 1043
Total cold run time: 116888 ms
Total hot run time: 38269 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4606 4313 4268 4268
q2 411 318 318 318
q3 2848 2677 2676 2676
q4 2023 1757 1773 1757
q5 5708 5762 5699 5699
q6 247 152 152 152
q7 2154 1785 1791 1785
q8 3292 3479 3446 3446
q9 8830 8820 8820 8820
q10 3602 3375 3495 3375
q11 618 523 516 516
q12 839 679 690 679
q13 17259 3167 3029 3029
q14 328 304 306 304
q15 581 539 534 534
q16 531 466 473 466
q17 1859 1531 1531 1531
q18 8326 7697 7606 7606
q19 6439 1616 1739 1616
q20 2189 1902 1879 1879
q21 13268 5295 5399 5295
q22 1183 1098 1089 1089
Total cold run time: 87141 ms
Total hot run time: 56840 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2304898032
<details>
<summary>TPC-H: <b>Total hot run time: 38564 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0b1e4a8f000eb4cf1faa30535e050f844fb38a05, data reload: false
------ Round 1 ----------------------------------
q1 18521 4576 4335 4335
q2 2038 187 180 180
q3 11866 1037 1103 1037
q4 10306 799 797 797
q5 7774 2856 2800 2800
q6 233 139 137 137
q7 993 620 611 611
q8 9342 2085 2093 2085
q9 7074 6528 6583 6528
q10 6997 2189 2170 2170
q11 461 252 244 244
q12 400 220 228 220
q13 17760 3093 3058 3058
q14 276 237 233 233
q15 526 493 498 493
q16 502 414 389 389
q17 999 716 692 692
q18 7482 6956 6910 6910
q19 1400 1100 1076 1076
q20 686 337 328 328
q21 3976 3312 3229 3229
q22 1123 1012 1036 1012
Total cold run time: 110735 ms
Total hot run time: 38564 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4468 4368 4330 4330
q2 382 284 275 275
q3 2896 2713 2662 2662
q4 1967 1724 1710 1710
q5 5718 5694 5844 5694
q6 227 137 142 137
q7 2302 1838 1825 1825
q8 3289 3473 3520 3473
q9 8803 8865 8885 8865
q10 3627 3422 3345 3345
q11 634 500 538 500
q12 854 703 656 656
q13 15429 3269 3317 3269
q14 333 285 293 285
q15 536 509 505 505
q16 504 440 456 440
q17 1841 1535 1534 1534
q18 8289 7847 7949 7847
q19 1734 1651 1630 1630
q20 2211 1898 1913 1898
q21 5934 5586 5467 5467
q22 1157 1052 1061 1052
Total cold run time: 73135 ms
Total hot run time: 57399 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2249275217
<details>
<summary>TPC-H: <b>Total hot run time: 40391 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7e2b1bd85dd3827c86fb9ead1aa462b016a3aba, data reload: false
------ Round 1 ----------------------------------
q1 18717 4643 4364 4364
q2 3105 197 207 197
q3 11146 1201 1118 1118
q4 10729 912 778 778
q5 8172 2730 2815 2730
q6 225 147 141 141
q7 985 624 597 597
q8 9222 2092 2099 2092
q9 8857 6571 6547 6547
q10 8787 3860 3764 3764
q11 466 238 236 236
q12 400 231 229 229
q13 17765 3019 3017 3017
q14 271 227 247 227
q15 516 486 500 486
q16 477 390 387 387
q17 976 690 650 650
q18 8162 7616 7586 7586
q19 6836 1477 1462 1462
q20 709 318 331 318
q21 4918 3274 3184 3184
q22 360 293 281 281
Total cold run time: 121801 ms
Total hot run time: 40391 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4458 4289 4290 4289
q2 367 269 266 266
q3 2996 2803 2694 2694
q4 1941 1599 1601 1599
q5 5291 5358 5332 5332
q6 223 131 135 131
q7 2176 1731 1724 1724
q8 3211 3373 3316 3316
q9 8447 8466 8401 8401
q10 3891 3762 3651 3651
q11 573 494 476 476
q12 793 664 591 591
q13 16348 2984 3023 2984
q14 288 269 272 269
q15 506 485 482 482
q16 484 419 422 419
q17 1781 1502 1501 1501
q18 7633 7493 7484 7484
q19 3342 1543 1580 1543
q20 2019 1787 1783 1783
q21 4925 4707 4860 4707
q22 576 516 502 502
Total cold run time: 72269 ms
Total hot run time: 54144 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2262170374
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2261786135
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2262321636
<details>
<summary>TPC-DS: <b>Total hot run time: 170685 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f40d3af909e3a4ce446ee5217827a6811bce34d1, data reload: false
query1 919 381 370 370
query2 6455 1708 1670 1670
query3 6649 224 221 221
query4 20265 17544 17285 17285
query5 3664 517 520 517
query6 284 172 178 172
query7 4598 305 305 305
query8 274 201 205 201
query9 8506 2369 2356 2356
query10 434 282 278 278
query11 10519 10016 10112 10016
query12 122 88 97 88
query13 1646 384 366 366
query14 8846 7889 7857 7857
query15 210 199 163 163
query16 6672 462 423 423
query17 943 586 542 542
query18 1888 287 277 277
query19 194 142 159 142
query20 94 85 84 84
query21 204 104 99 99
query22 4261 4124 3928 3928
query23 34031 33736 33739 33736
query24 10288 3146 3129 3129
query25 679 415 403 403
query26 1706 154 157 154
query27 3160 282 285 282
query28 7380 2025 1997 1997
query29 1187 441 447 441
query30 240 155 154 154
query31 955 791 808 791
query32 102 59 57 57
query33 672 328 329 328
query34 913 497 503 497
query35 855 736 773 736
query36 1054 890 891 890
query37 272 83 81 81
query38 2987 2849 2781 2781
query39 861 805 855 805
query40 249 118 111 111
query41 49 45 45 45
query42 130 97 100 97
query43 469 434 414 414
query44 1167 712 744 712
query45 215 185 180 180
query46 1102 823 768 768
query47 1816 1739 1711 1711
query48 360 298 287 287
query49 924 420 403 403
query50 910 429 424 424
query51 6840 6802 6678 6678
query52 95 88 88 88
query53 251 181 184 181
query54 621 456 441 441
query55 77 74 74 74
query56 273 266 258 258
query57 1124 1076 1041 1041
query58 278 257 299 257
query59 2764 2382 2281 2281
query60 302 261 275 261
query61 96 94 94 94
query62 887 667 677 667
query63 211 188 186 186
query64 5712 1902 1899 1899
query65 3193 3097 3080 3080
query66 1321 331 334 331
query67 15391 14910 14715 14715
query68 5484 560 575 560
query69 701 379 307 307
query70 1109 1096 1044 1044
query71 468 282 272 272
query72 7551 2732 2504 2504
query73 929 333 326 326
query74 6115 5737 5697 5697
query75 3620 2735 2725 2725
query76 3217 1367 1395 1367
query77 538 301 302 301
query78 9488 8883 8886 8883
query79 1686 533 528 528
query80 1908 507 491 491
query81 570 225 225 225
query82 478 132 127 127
query83 275 172 165 165
query84 277 77 77 77
query85 1162 355 426 355
query86 448 308 309 308
query87 3321 3168 3133 3133
query88 3691 2384 2386 2384
query89 379 284 291 284
query90 1688 186 186 186
query91 125 99 99 99
query92 65 51 50 50
query93 2016 621 632 621
query94 807 286 297 286
query95 381 270 266 266
query96 595 280 276 276
query97 3207 3115 3059 3059
query98 246 204 196 196
query99 1628 1291 1277 1277
Total cold run time: 265989 ms
Total hot run time: 170685 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1708566284
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -387,10 +410,28 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
msg = "malformed version value";
break;
}
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
+ std::tie(code, msg) = task->wait();
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "wait txn lazy commit failed, txn_id="
+ << version_pb.txn_id();
+ break;
+ }
+ goto TRY_AGAIN;
+ }
response->add_versions(version_pb.version());
response->add_version_update_time_ms(version_pb.update_time_ms());
}
}
+ if (code != MetaServiceCode::OK) {
+ break;
+ }
+ }
+ if (code != MetaServiceCode::OK) {
+ break;
Review Comment:
> Delete the useless code
done
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -387,10 +410,28 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
msg = "malformed version value";
break;
}
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
+ std::tie(code, msg) = task->wait();
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "wait txn lazy commit failed, txn_id="
+ << version_pb.txn_id();
+ break;
+ }
+ goto TRY_AGAIN;
Review Comment:
> We don't need to retry again, just like the `get_version`, returning the version directly is enough.
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2275070120
<details>
<summary>TPC-H: <b>Total hot run time: 39821 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1ac8d3c9b6c996a098b4d7ff79645541deefde05, data reload: false
------ Round 1 ----------------------------------
q1 17670 5316 4296 4296
q2 2031 185 181 181
q3 10521 1175 1095 1095
q4 10190 811 705 705
q5 7511 2527 2510 2510
q6 218 138 135 135
q7 974 590 593 590
q8 9210 1920 1931 1920
q9 8820 6595 6591 6591
q10 7051 2166 2143 2143
q11 525 245 252 245
q12 390 219 221 219
q13 17946 2947 2972 2947
q14 293 247 232 232
q15 527 464 464 464
q16 643 519 526 519
q17 985 709 653 653
q18 8301 7503 7473 7473
q19 3272 997 1128 997
q20 769 411 422 411
q21 5387 4586 4476 4476
q22 1099 1019 1020 1019
Total cold run time: 114333 ms
Total hot run time: 39821 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4394 4297 4293 4293
q2 394 284 281 281
q3 2890 2670 2636 2636
q4 1906 1604 1631 1604
q5 5251 5317 5319 5317
q6 222 136 130 130
q7 2070 1699 1676 1676
q8 3161 3310 3361 3310
q9 8425 8441 8406 8406
q10 3376 3101 3175 3101
q11 605 514 513 513
q12 782 633 585 585
q13 17639 2979 2963 2963
q14 325 268 274 268
q15 513 474 474 474
q16 620 551 549 549
q17 1787 1493 1485 1485
q18 7797 7648 7299 7299
q19 1710 1605 1685 1605
q20 2095 1881 1873 1873
q21 5390 5174 5212 5174
q22 1070 1021 1012 1012
Total cold run time: 72422 ms
Total hot run time: 54554 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2285272777
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2297034258
<details>
<summary>TPC-H: <b>Total hot run time: 38448 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26c7330ea2ca69f58795f6cc0b7821111e82c0d5, data reload: false
------ Round 1 ----------------------------------
q1 18361 4986 4315 4315
q2 2059 215 242 215
q3 11074 964 1081 964
q4 10525 803 788 788
q5 7791 2887 2843 2843
q6 266 161 162 161
q7 1042 656 664 656
q8 9568 2136 2119 2119
q9 8660 6596 6623 6596
q10 7077 2353 2254 2254
q11 529 274 279 274
q12 431 257 264 257
q13 17794 3010 3001 3001
q14 310 268 257 257
q15 557 536 537 536
q16 538 431 420 420
q17 1000 687 716 687
q18 7578 6773 6898 6773
q19 6459 1121 1127 1121
q20 727 352 365 352
q21 3870 2984 2827 2827
q22 1143 1054 1032 1032
Total cold run time: 117359 ms
Total hot run time: 38448 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4605 4295 4294 4294
q2 420 323 322 322
q3 2881 2649 2747 2649
q4 1989 1773 1724 1724
q5 5634 5774 5637 5637
q6 253 152 153 152
q7 2137 1768 1736 1736
q8 3302 3527 3457 3457
q9 8759 8721 8728 8721
q10 3581 3357 3327 3327
q11 599 528 528 528
q12 838 684 633 633
q13 17174 3181 3174 3174
q14 322 298 296 296
q15 546 523 508 508
q16 491 469 467 467
q17 1871 1552 1525 1525
q18 8197 8036 7874 7874
q19 7830 1645 1628 1628
q20 2209 1916 1884 1884
q21 14189 5365 5354 5354
q22 1203 1098 1063 1063
Total cold run time: 89030 ms
Total hot run time: 56953 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1730626928
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
- // Save table versions
- for (auto& i : table_id_tablet_ids) {
- std::string ver_key = table_version_key({instance_id, db_id, i.first});
- txn->atomic_add(ver_key, 1);
- LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key) << " txn_id=" << txn_id;
- }
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << "txn_id=" << txn_id << " table_id=" << table_id
+ << " partition_id=" << partition_id << " version=" << i.second;
- LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
- // Update txn_info
- txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- auto now_time = system_clock::now();
- uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
- if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
- LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
- << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
- return;
- }
- txn_info.set_commit_time(commit_time);
- txn_info.set_finish_time(commit_time);
- if (request->has_commit_attachment()) {
- txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
- }
- LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
- info_val.clear();
- if (!txn_info.SerializeToString(&info_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(info_key, info_val);
- LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+ for (size_t i = 0; i < lock_keys.size(); i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- // Update stats of affected tablet
- for (auto& [tablet_id, stats] : tablet_stats) {
- DCHECK(tablet_ids.count(tablet_id));
- auto& tablet_idx = tablet_ids[tablet_id];
- StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
- tablet_idx.partition_id(), tablet_id};
- update_tablet_stats(info, stats, txn, code, msg);
- if (code != MetaServiceCode::OK) return;
- }
- // Remove tmp rowset meta
- for (auto& [k, _] : tmp_rowsets_meta) {
- txn->remove(k);
- LOG(INFO) << "xxx remove tmp_rowset_key=" << hex(k) << " txn_id=" << txn_id;
- }
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
- const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
- LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
- txn->remove(running_key);
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
- std::string recycle_val;
- std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
- RecycleTxnPB recycle_pb;
- recycle_pb.set_creation_time(commit_time);
- recycle_pb.set_label(txn_info.label());
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
+ << " txn_id=" << txn_id;
+ }
- if (!recycle_pb.SerializeToString(&recycle_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(recycle_key, recycle_val);
+ LOG(INFO) << "commit_txn_eventually put_size=" << txn->put_bytes()
+ << " del_size=" << txn->delete_bytes() << " num_put_keys=" << txn->num_put_keys()
+ << " num_del_keys=" << txn->num_del_keys()
+ << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
- if (txn_info.load_job_source_type() ==
- LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
- put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
- }
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
- LOG(INFO) << "xxx commit_txn put recycle_key key=" << hex(recycle_key) << " txn_id=" << txn_id;
- LOG(INFO) << "commit_txn put_size=" << txn->put_bytes() << " del_size=" << txn->delete_bytes()
- << " num_put_keys=" << txn->num_put_keys() << " num_del_keys=" << txn->num_del_keys()
- << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer->submit(instance_id, txn_id);
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ if (ret.first != MetaServiceCode::OK) {
+ LOG(WARNING) << "lazy commit txn failed txn_id=" << txn_id << " code=" << ret.first
Review Comment:
add more info in log to describe what it will do if st is not OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732718352
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
- // Save table versions
- for (auto& i : table_id_tablet_ids) {
- std::string ver_key = table_version_key({instance_id, db_id, i.first});
- txn->atomic_add(ver_key, 1);
- LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key) << " txn_id=" << txn_id;
- }
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << "txn_id=" << txn_id << " table_id=" << table_id
+ << " partition_id=" << partition_id << " version=" << i.second;
- LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
- // Update txn_info
- txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- auto now_time = system_clock::now();
- uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
- if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
- LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
- << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
- return;
- }
- txn_info.set_commit_time(commit_time);
- txn_info.set_finish_time(commit_time);
- if (request->has_commit_attachment()) {
- txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
- }
- LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
- info_val.clear();
- if (!txn_info.SerializeToString(&info_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(info_key, info_val);
- LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+ for (size_t i = 0; i < lock_keys.size(); i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- // Update stats of affected tablet
- for (auto& [tablet_id, stats] : tablet_stats) {
- DCHECK(tablet_ids.count(tablet_id));
- auto& tablet_idx = tablet_ids[tablet_id];
- StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
- tablet_idx.partition_id(), tablet_id};
- update_tablet_stats(info, stats, txn, code, msg);
- if (code != MetaServiceCode::OK) return;
- }
- // Remove tmp rowset meta
- for (auto& [k, _] : tmp_rowsets_meta) {
- txn->remove(k);
- LOG(INFO) << "xxx remove tmp_rowset_key=" << hex(k) << " txn_id=" << txn_id;
- }
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
- const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
- LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
- txn->remove(running_key);
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
- std::string recycle_val;
- std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
- RecycleTxnPB recycle_pb;
- recycle_pb.set_creation_time(commit_time);
- recycle_pb.set_label(txn_info.label());
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
+ << " txn_id=" << txn_id;
+ }
- if (!recycle_pb.SerializeToString(&recycle_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(recycle_key, recycle_val);
+ LOG(INFO) << "commit_txn_eventually put_size=" << txn->put_bytes()
+ << " del_size=" << txn->delete_bytes() << " num_put_keys=" << txn->num_put_keys()
+ << " num_del_keys=" << txn->num_del_keys()
+ << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
- if (txn_info.load_job_source_type() ==
- LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
- put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
- }
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
- LOG(INFO) << "xxx commit_txn put recycle_key key=" << hex(recycle_key) << " txn_id=" << txn_id;
- LOG(INFO) << "commit_txn put_size=" << txn->put_bytes() << " del_size=" << txn->delete_bytes()
- << " num_put_keys=" << txn->num_put_keys() << " num_del_keys=" << txn->num_del_keys()
- << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer->submit(instance_id, txn_id);
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ if (ret.first != MetaServiceCode::OK) {
+ LOG(WARNING) << "lazy commit txn failed txn_id=" << txn_id << " code=" << ret.first
+ << "msg=" << ret.second;
+ }
- // Finally we are done...
- err = txn->commit();
- if (err != TxnErrorCode::TXN_OK) {
- if ((err == TxnErrorCode::TXN_VALUE_TOO_LARGE ||
- err == TxnErrorCode::TXN_BYTES_TOO_LARGE) &&
- !tmp_rowsets_meta.empty()) {
- size_t max_size = 0, max_idx = 0, max_num_segments = 0,
- min_num_segments = std::numeric_limits<size_t>::max(), avg_num_segments = 0;
- for (size_t i = 0; i < tmp_rowsets_meta.size(); i++) {
- auto& [k, v] = tmp_rowsets_meta[i];
- if (v.ByteSizeLong() > max_size) {
- max_size = v.ByteSizeLong();
- max_idx = i;
- }
- if (v.num_segments() > max_num_segments) {
- max_num_segments = v.num_segments();
- }
- if (v.num_segments() < min_num_segments) {
- min_num_segments = v.num_segments();
- }
- avg_num_segments += v.num_segments();
- }
- if (!tmp_rowsets_meta.empty()) {
- avg_num_segments /= tmp_rowsets_meta.size();
- }
- LOG(WARNING) << "failed to commit kv txn"
- << ", err=" << err << ", txn_id=" << txn_id
- << ", total_rowsets=" << tmp_rowsets_meta.size()
- << ", avg_num_segments=" << avg_num_segments
- << ", min_num_segments=" << min_num_segments
- << ", max_num_segments=" << max_num_segments
- << ", largest_rowset_size=" << max_size
- << ", largest_rowset_key=" << hex(tmp_rowsets_meta[max_idx].first)
- << ", largest_rowset_value="
- << tmp_rowsets_meta[max_idx].second.ShortDebugString();
+ std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ // Accumulate affected rows
+ auto& stats = tablet_stats[i.tablet_id()];
+ stats.data_size += i.data_disk_size();
+ stats.num_rows += i.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += i.num_segments();
}
- code = cast_as<ErrCategory::COMMIT>(err);
- ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- return;
- }
- // calculate table stats from tablets stats
- std::map<int64_t /*table_id*/, TableStats> table_stats;
- std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
- request->base_tablet_ids().end());
- calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
- for (const auto& pair : table_stats) {
- TableStatsPB* stats_pb = response->add_table_stats();
- auto table_id = pair.first;
- auto stats = pair.second;
- get_pb_from_tablestats(stats, stats_pb);
- stats_pb->set_table_id(table_id);
- VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
- << " table_id=" << table_id
- << " updated_row_count=" << stats_pb->updated_row_count();
- }
+ // calculate table stats from tablets stats
+ std::map<int64_t /*table_id*/, TableStats> table_stats;
+ std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
+ request->base_tablet_ids().end());
+ calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
+ for (const auto& pair : table_stats) {
+ TableStatsPB* stats_pb = response->add_table_stats();
+ auto table_id = pair.first;
+ auto stats = pair.second;
+ get_pb_from_tablestats(stats, stats_pb);
+ stats_pb->set_table_id(table_id);
+ VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
+ << " table_id=" << table_id
+ << " updated_row_count=" << stats_pb->updated_row_count();
+ }
- response->mutable_txn_info()->CopyFrom(txn_info);
-} // end commit_txn
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
Review Comment:
> is it VISIBLE even if `task->wait()` failed?
yes, committed txn must be advanced
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2314649999
<details>
<summary>TPC-DS: <b>Total hot run time: 192851 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0f6412f7fa9a90fc50e6af06a04d4e9b42655491, data reload: false
query1 1246 897 880 880
query2 6377 1979 1930 1930
query3 10770 4109 4024 4024
query4 59530 23957 23403 23403
query5 5439 497 493 493
query6 407 166 159 159
query7 5847 298 305 298
query8 292 204 220 204
query9 9045 2477 2491 2477
query10 501 272 262 262
query11 16900 15114 15206 15114
query12 159 109 106 106
query13 1623 400 396 396
query14 11171 7094 7481 7094
query15 234 171 183 171
query16 7618 447 477 447
query17 1157 551 558 551
query18 2098 307 298 298
query19 285 150 149 149
query20 122 115 110 110
query21 205 105 105 105
query22 4664 4582 4606 4582
query23 34223 33521 33123 33123
query24 5927 2826 2825 2825
query25 527 385 388 385
query26 683 150 151 150
query27 1811 272 274 272
query28 3816 2114 2100 2100
query29 690 408 407 407
query30 235 155 154 154
query31 927 725 782 725
query32 88 57 62 57
query33 471 300 291 291
query34 870 480 486 480
query35 863 725 723 723
query36 1057 902 960 902
query37 152 93 96 93
query38 3907 3845 3922 3845
query39 1457 1372 1394 1372
query40 193 114 111 111
query41 47 48 45 45
query42 112 94 93 93
query43 516 486 481 481
query44 1077 743 756 743
query45 194 163 163 163
query46 1086 743 757 743
query47 1906 1801 1833 1801
query48 359 287 300 287
query49 752 415 429 415
query50 818 418 409 409
query51 7243 7063 7089 7063
query52 95 87 84 84
query53 252 182 180 180
query54 556 446 452 446
query55 81 80 76 76
query56 281 241 241 241
query57 1208 1083 1062 1062
query58 224 222 225 222
query59 2970 2870 2743 2743
query60 305 268 260 260
query61 103 95 97 95
query62 736 664 643 643
query63 214 179 179 179
query64 2835 657 686 657
query65 3213 3161 3132 3132
query66 663 328 327 327
query67 15548 15320 15191 15191
query68 2966 569 576 569
query69 407 274 289 274
query70 1223 1089 1074 1074
query71 354 278 266 266
query72 5169 4035 3932 3932
query73 745 332 332 332
query74 9089 8856 8846 8846
query75 3351 2647 2671 2647
query76 1465 1009 1033 1009
query77 544 317 313 313
query78 9706 9374 10068 9374
query79 1088 532 538 532
query80 678 498 504 498
query81 463 237 228 228
query82 247 148 145 145
query83 172 153 148 148
query84 256 74 77 74
query85 675 271 361 271
query86 304 303 287 287
query87 4461 4199 4299 4199
query88 3296 2312 2337 2312
query89 386 282 281 281
query90 1932 194 189 189
query91 121 98 99 98
query92 60 55 52 52
query93 1049 536 542 536
query94 692 287 304 287
query95 329 258 267 258
query96 595 270 273 270
query97 3204 3041 3044 3041
query98 212 197 198 197
query99 1500 1315 1263 1263
Total cold run time: 303720 ms
Total hot run time: 192851 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732025419
##########
cloud/src/common/config.h:
##########
@@ -206,4 +206,8 @@ CONF_Validator(s3_client_http_scheme, [](const std::string& config) -> bool {
// Max retry times for object storage request
CONF_mInt64(max_s3_client_retry, "10");
+CONF_Bool(enable_txn_lazy_commit, "true");
+CONF_Int32(txn_lazy_commit_rowsets_thresold, "2048");
+CONF_Int32(txn_lazy_commit_num_threads, "8");
+CONF_Int32(txn_lazy_max_rowsets_per_batch, "2048");
Review Comment:
> what the relation between `txn_lazy_max_rowsets_per_batch` and `txn_lazy_max_rowsets_per_batch`
"txn_lazy_max_rowsets_per_batch" the max num for convert tmp rowsets
"txn_lazy_commit_rowsets_thresold" the num to determine whether lazy commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2303643068
<details>
<summary>TPC-H: <b>Total hot run time: 39019 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 740144c09664bd5fcc4a77551e5f4fdf659aa12d, data reload: false
------ Round 1 ----------------------------------
q1 18495 4994 4431 4431
q2 2069 219 225 219
q3 11493 1004 1169 1004
q4 10379 752 801 752
q5 7761 2931 2869 2869
q6 272 160 160 160
q7 1030 658 651 651
q8 9414 2172 2176 2172
q9 7223 6615 6629 6615
q10 7094 2186 2266 2186
q11 499 277 278 277
q12 439 269 265 265
q13 18948 3056 3149 3056
q14 318 257 275 257
q15 557 519 520 519
q16 535 422 412 412
q17 1027 749 803 749
q18 7547 6838 7078 6838
q19 1444 1054 1052 1052
q20 728 359 361 359
q21 4234 3144 3457 3144
q22 1131 1069 1032 1032
Total cold run time: 112637 ms
Total hot run time: 39019 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4443 4404 4343 4343
q2 415 297 308 297
q3 2897 2661 2799 2661
q4 2012 1791 1725 1725
q5 5715 5693 5832 5693
q6 242 150 146 146
q7 2279 1872 1835 1835
q8 3330 3484 3532 3484
q9 8901 8900 8795 8795
q10 3590 3442 3420 3420
q11 645 555 533 533
q12 847 670 702 670
q13 16021 3238 3254 3238
q14 339 300 301 300
q15 573 522 530 522
q16 514 478 477 477
q17 1888 1573 1559 1559
q18 8231 7971 7940 7940
q19 1805 1618 1495 1495
q20 2136 1942 1959 1942
q21 5669 5581 5379 5379
q22 1205 1130 1103 1103
Total cold run time: 73697 ms
Total hot run time: 57557 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2301568583
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2285789380
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1714982367
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:292:** 122 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
Review Comment:
warning: function 'convert_tmp_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void convert_tmp_rowsets(
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:40:** 152 lines including whitespace and comments (threshold 80)
```cpp
void convert_tmp_rowsets(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312273940
<details>
<summary>ClickBench: <b>Total hot run time: 30.49 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb825b12499087f8dea29592d393d5582d3623c3, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.04 0.04
query3 0.22 0.05 0.05
query4 1.67 0.07 0.08
query5 0.50 0.49 0.51
query6 1.13 0.74 0.73
query7 0.02 0.01 0.02
query8 0.05 0.05 0.05
query9 0.54 0.48 0.49
query10 0.55 0.55 0.54
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.60 0.61 0.59
query14 0.75 0.79 0.79
query15 0.85 0.82 0.84
query16 0.36 0.38 0.35
query17 1.05 1.01 1.06
query18 0.22 0.20 0.21
query19 1.89 1.74 1.87
query20 0.02 0.01 0.01
query21 15.43 0.66 0.65
query22 4.51 6.35 1.88
query23 18.26 1.33 1.26
query24 2.08 0.23 0.21
query25 0.15 0.08 0.08
query26 0.26 0.17 0.18
query27 0.08 0.08 0.07
query28 13.27 1.01 0.99
query29 12.63 3.27 3.22
query30 0.24 0.06 0.06
query31 2.88 0.40 0.39
query32 3.23 0.47 0.48
query33 3.00 3.00 3.04
query34 17.27 4.39 4.37
query35 4.45 4.37 4.45
query36 0.66 0.45 0.49
query37 0.18 0.15 0.16
query38 0.15 0.16 0.16
query39 0.05 0.04 0.04
query40 0.16 0.14 0.13
query41 0.10 0.05 0.04
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.98 s
Total hot run time: 30.49 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #38243:
URL: https://github.com/apache/doris/pull/38243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2314491020
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732724162
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
Review Comment:
> what if the commit returns errors like
>
> 1. value exceeds max bytes
> 2. transaction too old
return error to ms client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2304159370
<details>
<summary>ClickBench: <b>Total hot run time: 31.27 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 20e25bf51ae7f01a6fff2f9704c155a8c059045a, data reload: false
query1 0.05 0.05 0.04
query2 0.08 0.04 0.04
query3 0.22 0.05 0.05
query4 1.66 0.08 0.08
query5 0.52 0.48 0.48
query6 1.13 0.75 0.72
query7 0.02 0.02 0.02
query8 0.06 0.04 0.04
query9 0.57 0.49 0.48
query10 0.55 0.54 0.54
query11 0.15 0.12 0.11
query12 0.16 0.12 0.12
query13 0.61 0.58 0.59
query14 0.78 0.76 0.81
query15 0.84 0.83 0.81
query16 0.37 0.36 0.36
query17 1.05 1.01 1.05
query18 0.22 0.20 0.21
query19 1.85 1.78 1.73
query20 0.01 0.02 0.01
query21 15.41 0.66 0.67
query22 3.95 7.00 2.54
query23 18.32 1.41 1.28
query24 2.17 0.22 0.21
query25 0.14 0.08 0.08
query26 0.27 0.18 0.18
query27 0.07 0.08 0.07
query28 13.22 1.02 1.00
query29 12.63 3.33 3.30
query30 0.24 0.06 0.06
query31 2.87 0.40 0.39
query32 3.29 0.48 0.48
query33 2.96 3.00 3.00
query34 17.12 4.36 4.41
query35 4.42 4.41 4.42
query36 0.65 0.48 0.49
query37 0.19 0.16 0.17
query38 0.16 0.14 0.16
query39 0.05 0.04 0.04
query40 0.15 0.12 0.12
query41 0.09 0.06 0.05
query42 0.06 0.05 0.04
query43 0.05 0.05 0.04
Total cold run time: 109.38 s
Total hot run time: 31.27 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2297056462
<details>
<summary>TPC-DS: <b>Total hot run time: 195462 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26c7330ea2ca69f58795f6cc0b7821111e82c0d5, data reload: false
query1 1328 904 896 896
query2 6630 2082 1939 1939
query3 10736 3805 3803 3803
query4 58330 25176 23404 23404
query5 5830 737 728 728
query6 542 209 209 209
query7 6403 332 336 332
query8 538 437 459 437
query9 9152 2537 2513 2513
query10 602 349 338 338
query11 18363 15018 15463 15018
query12 227 151 141 141
query13 1688 438 472 438
query14 11553 7240 6909 6909
query15 246 194 208 194
query16 7668 522 537 522
query17 1201 638 623 623
query18 2129 346 336 336
query19 302 170 165 165
query20 154 138 139 138
query21 245 166 148 148
query22 4482 4352 4634 4352
query23 34193 33918 33690 33690
query24 5702 3030 3077 3030
query25 588 439 449 439
query26 717 185 205 185
query27 1861 300 314 300
query28 4252 2172 2152 2152
query29 731 452 453 452
query30 232 184 186 184
query31 1024 847 819 819
query32 98 77 79 77
query33 499 359 339 339
query34 890 511 494 494
query35 859 795 773 773
query36 1062 979 953 953
query37 156 103 106 103
query38 3998 3837 3781 3781
query39 1525 1472 1457 1457
query40 238 158 157 157
query41 138 135 135 135
query42 137 117 115 115
query43 561 511 507 507
query44 1154 784 795 784
query45 227 196 196 196
query46 1114 779 747 747
query47 1909 1821 1854 1821
query48 419 338 333 333
query49 905 583 589 583
query50 858 466 476 466
query51 6890 6770 6722 6722
query52 126 111 108 108
query53 304 225 244 225
query54 628 529 492 492
query55 91 88 91 88
query56 339 330 315 315
query57 1224 1135 1128 1128
query58 307 313 296 296
query59 3016 2878 2833 2833
query60 356 331 330 330
query61 152 146 145 145
query62 796 712 694 694
query63 262 229 228 228
query64 3426 1841 1822 1822
query65 3237 3182 3203 3182
query66 1022 732 670 670
query67 15239 15049 14920 14920
query68 4918 564 575 564
query69 536 318 311 311
query70 1222 1152 1171 1152
query71 496 315 305 305
query72 7494 2272 2055 2055
query73 795 351 362 351
query74 9491 8754 8781 8754
query75 3529 2744 2726 2726
query76 2764 1033 1036 1033
query77 873 430 433 430
query78 9632 9177 9069 9069
query79 7151 561 550 550
query80 1131 599 611 599
query81 606 260 254 254
query82 917 164 158 158
query83 419 209 208 208
query84 307 94 105 94
query85 983 355 352 352
query86 399 317 322 317
query87 4362 4123 4200 4123
query88 4584 2464 2507 2464
query89 444 318 326 318
query90 2137 229 222 222
query91 151 130 125 125
query92 89 74 73 73
query93 5986 539 554 539
query94 938 307 318 307
query95 385 297 293 293
query96 635 281 287 281
query97 3218 3069 3081 3069
query98 246 225 218 218
query99 1723 1295 1311 1295
Total cold run time: 330372 ms
Total hot run time: 195462 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2284304022
<details>
<summary>TPC-H: <b>Total hot run time: 39994 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1ee23142cdd036222be1b6f5f0645252c6951fd9, data reload: false
------ Round 1 ----------------------------------
q1 18257 4662 4393 4393
q2 2729 185 192 185
q3 11755 1154 1167 1154
q4 10660 748 670 670
q5 8234 2615 2530 2530
q6 232 141 144 141
q7 992 588 594 588
q8 9223 1912 1909 1909
q9 8713 6574 6571 6571
q10 7044 2254 2251 2251
q11 470 240 241 240
q12 390 221 221 221
q13 17753 2954 2964 2954
q14 284 239 237 237
q15 524 482 464 464
q16 631 521 521 521
q17 977 765 779 765
q18 8101 7432 7505 7432
q19 5698 1078 1139 1078
q20 741 423 422 422
q21 5374 4285 4708 4285
q22 1085 1042 983 983
Total cold run time: 119867 ms
Total hot run time: 39994 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4532 4273 4265 4265
q2 380 274 273 273
q3 2862 2619 2612 2612
q4 1862 1604 1579 1579
q5 5276 5220 5277 5220
q6 225 133 131 131
q7 2027 1625 1659 1625
q8 3166 3343 3331 3331
q9 8451 8327 8317 8317
q10 3363 3116 3099 3099
q11 594 475 490 475
q12 782 599 603 599
q13 16979 2980 2957 2957
q14 292 264 269 264
q15 509 488 477 477
q16 598 571 559 559
q17 1793 1481 1473 1473
q18 7725 7509 7493 7493
q19 1670 1473 1520 1473
q20 2113 1916 1896 1896
q21 5132 5042 5140 5042
q22 1089 1001 1003 1001
Total cold run time: 71420 ms
Total hot run time: 54161 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2285206704
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312205190
<details>
<summary>TPC-H: <b>Total hot run time: 38102 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb825b12499087f8dea29592d393d5582d3623c3, data reload: false
------ Round 1 ----------------------------------
q1 18034 5219 4293 4293
q2 2026 184 182 182
q3 11744 970 1178 970
q4 10524 793 776 776
q5 7793 2912 2760 2760
q6 222 134 134 134
q7 964 621 608 608
q8 9325 2095 2093 2093
q9 7048 6537 6570 6537
q10 7015 2212 2232 2212
q11 473 242 245 242
q12 402 219 221 219
q13 17757 3071 3009 3009
q14 282 237 235 235
q15 521 486 494 486
q16 497 389 393 389
q17 995 677 667 667
q18 7490 6802 6843 6802
q19 1395 1020 985 985
q20 681 331 333 331
q21 3915 3167 3151 3151
q22 1098 1021 1031 1021
Total cold run time: 110201 ms
Total hot run time: 38102 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4321 4292 4254 4254
q2 390 261 284 261
q3 2929 2672 2634 2634
q4 1992 1630 1635 1630
q5 5726 5722 5779 5722
q6 239 139 139 139
q7 2217 1862 1830 1830
q8 3320 3452 3543 3452
q9 8890 8881 8833 8833
q10 3608 3382 3356 3356
q11 620 500 500 500
q12 817 671 645 645
q13 16458 3068 3250 3068
q14 325 287 287 287
q15 544 514 494 494
q16 503 446 436 436
q17 1830 1513 1546 1513
q18 8191 7944 7995 7944
q19 1712 1527 1604 1527
q20 2159 1899 1898 1898
q21 5821 5470 5641 5470
q22 1153 1009 1010 1009
Total cold run time: 73765 ms
Total hot run time: 56902 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312960925
<details>
<summary>TPC-DS: <b>Total hot run time: 192287 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40557e2131ead7df254d700743017f557f0777e6, data reload: false
query1 1265 891 888 888
query2 6329 1912 1920 1912
query3 10584 3856 3863 3856
query4 59555 27470 23284 23284
query5 5387 499 510 499
query6 404 163 159 159
query7 5786 295 296 295
query8 280 212 216 212
query9 9076 2487 2496 2487
query10 505 267 263 263
query11 18530 14935 15357 14935
query12 155 99 101 99
query13 1568 388 381 381
query14 10750 7337 7335 7335
query15 220 172 176 172
query16 7575 477 470 470
query17 1136 586 585 585
query18 2048 312 304 304
query19 288 151 154 151
query20 119 107 111 107
query21 206 104 106 104
query22 4615 4410 4377 4377
query23 34423 33096 33491 33096
query24 6026 2825 2860 2825
query25 531 391 379 379
query26 689 155 154 154
query27 1767 284 277 277
query28 3749 2055 2045 2045
query29 691 401 402 401
query30 237 152 161 152
query31 963 755 768 755
query32 85 54 56 54
query33 468 289 275 275
query34 867 480 481 480
query35 840 698 715 698
query36 1087 922 932 922
query37 138 85 79 79
query38 3983 3863 3937 3863
query39 1455 1377 1497 1377
query40 198 116 111 111
query41 46 45 44 44
query42 119 97 96 96
query43 494 471 472 471
query44 1086 749 755 749
query45 193 168 165 165
query46 1105 759 777 759
query47 1878 1800 1806 1800
query48 376 292 295 292
query49 767 439 443 439
query50 826 427 414 414
query51 7066 7099 7031 7031
query52 101 87 88 87
query53 253 181 192 181
query54 570 477 468 468
query55 77 78 79 78
query56 302 271 274 271
query57 1188 1094 1042 1042
query58 230 234 229 229
query59 3148 2747 2942 2747
query60 310 273 290 273
query61 131 119 123 119
query62 778 667 669 667
query63 225 184 190 184
query64 2928 766 726 726
query65 3246 3167 3143 3143
query66 699 347 359 347
query67 15335 15116 15146 15116
query68 3494 586 562 562
query69 414 276 281 276
query70 1200 1088 1121 1088
query71 364 279 276 276
query72 6499 4034 3934 3934
query73 750 336 345 336
query74 9482 8793 8900 8793
query75 3364 2683 2717 2683
query76 1499 925 948 925
query77 530 327 328 327
query78 10338 9128 9029 9029
query79 1086 546 544 544
query80 874 499 524 499
query81 556 228 225 225
query82 239 135 136 135
query83 224 147 148 147
query84 261 77 75 75
query85 836 285 283 283
query86 364 293 298 293
query87 4387 4317 4202 4202
query88 3184 2386 2370 2370
query89 400 291 281 281
query90 1997 192 195 192
query91 122 100 100 100
query92 63 53 55 53
query93 1882 541 548 541
query94 778 285 306 285
query95 347 272 268 268
query96 642 278 274 274
query97 3203 3078 3075 3075
query98 220 202 203 202
query99 1577 1258 1299 1258
Total cold run time: 309231 ms
Total hot run time: 192287 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2245098501
<details>
<summary>TPC-DS: <b>Total hot run time: 174099 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d47f33a15e68079183da96a15302b529dc1c36a2, data reload: false
query1 910 371 363 363
query2 6448 1891 1811 1811
query3 6647 208 218 208
query4 25909 17478 17392 17392
query5 3724 479 493 479
query6 278 185 178 178
query7 4592 293 298 293
query8 253 206 199 199
query9 8468 2457 2417 2417
query10 444 307 260 260
query11 10654 10192 10091 10091
query12 115 91 88 88
query13 1637 373 365 365
query14 10227 7665 7243 7243
query15 224 165 167 165
query16 7511 458 493 458
query17 1130 547 530 530
query18 1362 277 266 266
query19 190 153 142 142
query20 87 82 81 81
query21 206 131 125 125
query22 4349 4098 4047 4047
query23 34032 33717 33673 33673
query24 10979 2981 2938 2938
query25 611 387 396 387
query26 709 152 158 152
query27 2188 278 280 278
query28 6181 2093 2104 2093
query29 884 661 648 648
query30 264 155 153 153
query31 994 753 759 753
query32 101 56 56 56
query33 657 338 336 336
query34 895 485 493 485
query35 876 748 731 731
query36 1133 958 972 958
query37 142 80 81 80
query38 2893 2882 2838 2838
query39 918 875 850 850
query40 225 126 130 126
query41 49 46 52 46
query42 108 105 104 104
query43 488 478 465 465
query44 1096 737 739 737
query45 202 169 167 167
query46 1098 790 719 719
query47 1919 1789 1826 1789
query48 368 289 286 286
query49 828 403 402 402
query50 780 386 386 386
query51 6863 6778 6631 6631
query52 101 90 87 87
query53 358 290 281 281
query54 877 451 446 446
query55 77 74 74 74
query56 283 263 267 263
query57 1145 1042 1074 1042
query58 248 253 249 249
query59 2786 2582 2573 2573
query60 310 272 282 272
query61 99 95 94 94
query62 806 640 627 627
query63 324 290 279 279
query64 9211 2192 1658 1658
query65 3169 3086 3131 3086
query66 745 325 333 325
query67 15638 15206 14964 14964
query68 4463 542 555 542
query69 455 331 339 331
query70 1213 1152 1116 1116
query71 369 276 271 271
query72 7272 5836 5618 5618
query73 762 329 327 327
query74 6198 5753 5724 5724
query75 3428 2668 2695 2668
query76 2234 921 1001 921
query77 468 295 298 295
query78 9537 9167 11057 9167
query79 3232 528 533 528
query80 1523 541 466 466
query81 596 221 225 221
query82 668 135 134 134
query83 290 167 165 165
query84 279 91 85 85
query85 1915 315 340 315
query86 477 325 310 310
query87 3307 3163 3122 3122
query88 4141 2394 2406 2394
query89 467 381 376 376
query90 1742 192 192 192
query91 127 101 102 101
query92 56 48 50 48
query93 3595 522 512 512
query94 1125 297 290 290
query95 416 314 333 314
query96 611 275 280 275
query97 3225 3033 3104 3033
query98 220 201 198 198
query99 1617 1263 1252 1252
Total cold run time: 276051 ms
Total hot run time: 174099 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2249218049
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1690977477
##########
cloud/src/meta-service/txn_lazy_committer.h:
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/cloud.pb.h>
Review Comment:
warning: 'gen_cpp/cloud.pb.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/cloud.pb.h>
^
```
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+extern void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+extern void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+extern void make_committed_txn_visible(const std::string& instance_id, int64_t db_id,
+ int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
Review Comment:
warning: function 'TxnLazyCommitTask' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:41:** 130 lines including whitespace and comments (threshold 80)
```cpp
TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2250325145
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2249285153
<details>
<summary>TPC-DS: <b>Total hot run time: 172476 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7e2b1bd85dd3827c86fb9ead1aa462b016a3aba, data reload: false
query1 920 379 371 371
query2 6462 1907 1803 1803
query3 6674 209 216 209
query4 24802 17722 17341 17341
query5 4173 481 495 481
query6 269 172 159 159
query7 4605 288 288 288
query8 244 197 198 197
query9 8453 2423 2398 2398
query10 454 308 295 295
query11 11869 10009 10177 10009
query12 134 87 80 80
query13 1631 372 368 368
query14 10274 6930 7830 6930
query15 226 176 171 171
query16 7840 484 473 473
query17 1579 586 553 553
query18 2007 287 292 287
query19 206 155 156 155
query20 89 85 84 84
query21 207 130 127 127
query22 4263 4156 4062 4062
query23 33805 33118 33094 33094
query24 11108 2884 2933 2884
query25 659 393 393 393
query26 1501 157 156 156
query27 2853 275 284 275
query28 7292 2029 2011 2011
query29 1004 645 644 644
query30 288 146 151 146
query31 971 733 756 733
query32 97 55 57 55
query33 793 359 341 341
query34 897 478 498 478
query35 847 741 736 736
query36 1102 917 934 917
query37 151 80 82 80
query38 2909 2801 2760 2760
query39 914 821 817 817
query40 290 121 122 121
query41 46 46 45 45
query42 125 106 102 102
query43 498 476 454 454
query44 1237 730 729 729
query45 203 174 175 174
query46 1097 718 745 718
query47 1886 1779 1805 1779
query48 370 293 305 293
query49 1178 435 421 421
query50 777 394 391 391
query51 6973 6628 6663 6628
query52 106 94 97 94
query53 361 293 297 293
query54 1047 457 460 457
query55 77 75 76 75
query56 309 293 295 293
query57 1184 1011 1031 1011
query58 263 253 282 253
query59 2886 2708 2559 2559
query60 314 275 279 275
query61 95 93 103 93
query62 850 643 671 643
query63 330 292 293 292
query64 10598 2222 1707 1707
query65 3169 3107 3096 3096
query66 1377 346 331 331
query67 15696 14833 14889 14833
query68 8798 548 542 542
query69 754 489 370 370
query70 1427 1147 1099 1099
query71 522 281 279 279
query72 9281 5564 5564 5564
query73 2262 326 326 326
query74 6088 5671 5619 5619
query75 5023 2711 2654 2654
query76 5417 978 960 960
query77 813 306 305 305
query78 10359 9366 8956 8956
query79 10962 543 538 538
query80 1076 479 489 479
query81 589 219 232 219
query82 339 139 132 132
query83 328 168 166 166
query84 274 83 86 83
query85 1029 315 306 306
query86 349 297 342 297
query87 3380 3195 3117 3117
query88 4900 2358 2370 2358
query89 513 386 392 386
query90 2125 196 196 196
query91 132 99 102 99
query92 60 48 48 48
query93 6731 532 525 525
query94 1458 300 296 296
query95 414 317 322 317
query96 622 271 275 271
query97 3189 3057 3000 3000
query98 228 208 195 195
query99 1582 1253 1271 1253
Total cold run time: 308938 ms
Total hot run time: 172476 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1708566029
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ break;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t, std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end = (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id_, txn_id_, txn_kv_, code_, msg_, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code_ != MetaServiceCode::OK) break;
+ }
+ if (code_ != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id_, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code_ = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id_
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id_ << " key=" << hex(ver_key);
+ msg_ = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id_) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id_;
+ msg_ = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
Review Comment:
> Handle `TxnErrorCode::TXN_CONFLICT` too.
Add a retry for lazy task `commit` method
##########
cloud/src/recycler/recycler_service.cpp:
##########
@@ -42,8 +42,12 @@ extern int reset_s3_rate_limiter(S3RateLimitType type, size_t max_speed, size_t
extern std::tuple<int, std::string_view> convert_ms_code_to_http_code(MetaServiceCode ret);
RecyclerServiceImpl::RecyclerServiceImpl(std::shared_ptr<TxnKv> txn_kv, Recycler* recycler,
- Checker* checker)
- : txn_kv_(std::move(txn_kv)), recycler_(recycler), checker_(checker) {}
+ Checker* checker,
+ std::shared_ptr<TxnLazyCommitter> txn_lazy_committer)
+ : txn_kv_(std::move(txn_kv)),
+ recycler_(recycler),
+ checker_(checker),
+ txn_lazy_committer_(txn_lazy_committer) {}
Review Comment:
>
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2297935290
<details>
<summary>TPC-DS: <b>Total hot run time: 196645 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43feec09f2a80e3fe7f3b07ebc1b5b49bdf0e0bf, data reload: false
query1 1365 911 893 893
query2 6614 1945 1962 1945
query3 10789 3964 3900 3900
query4 56317 23837 23261 23261
query5 5707 705 707 705
query6 544 215 232 215
query7 6347 321 319 319
query8 546 438 443 438
query9 9007 2531 2519 2519
query10 575 341 329 329
query11 17100 15216 15086 15086
query12 205 144 165 144
query13 1645 455 423 423
query14 11458 7570 7320 7320
query15 288 191 194 191
query16 7595 547 532 532
query17 1154 640 622 622
query18 1984 385 369 369
query19 329 170 171 170
query20 146 137 139 137
query21 251 143 143 143
query22 4602 4636 4384 4384
query23 34553 33655 34015 33655
query24 5696 2957 2989 2957
query25 578 435 432 432
query26 702 179 179 179
query27 1746 311 310 310
query28 3793 2187 2163 2163
query29 722 472 449 449
query30 230 189 201 189
query31 1063 879 830 830
query32 100 84 83 83
query33 510 358 344 344
query34 904 508 492 492
query35 841 787 785 785
query36 1094 966 954 954
query37 159 103 99 99
query38 3967 3904 3789 3789
query39 1532 1474 1473 1473
query40 232 155 154 154
query41 141 158 140 140
query42 144 118 116 116
query43 523 501 507 501
query44 1125 776 791 776
query45 228 195 197 195
query46 1121 764 771 764
query47 1986 1902 1887 1887
query48 418 344 334 334
query49 922 584 584 584
query50 860 460 469 460
query51 6791 6810 6865 6810
query52 127 107 115 107
query53 297 225 227 225
query54 614 510 501 501
query55 90 89 89 89
query56 334 320 306 306
query57 1224 1156 1187 1156
query58 302 302 299 299
query59 2952 2757 2908 2757
query60 346 324 324 324
query61 152 145 146 145
query62 775 711 719 711
query63 255 230 230 230
query64 3339 1922 1887 1887
query65 3252 3199 3183 3183
query66 1034 692 678 678
query67 15579 15088 15010 15010
query68 6810 584 616 584
query69 605 425 328 328
query70 1179 1111 1166 1111
query71 501 313 310 310
query72 6713 2277 2090 2090
query73 809 353 354 353
query74 9166 8853 8851 8851
query75 3742 2780 2766 2766
query76 3464 1039 1011 1011
query77 675 431 436 431
query78 10075 9111 9229 9111
query79 5121 548 552 548
query80 2623 604 602 602
query81 619 259 264 259
query82 856 162 159 159
query83 359 217 215 215
query84 303 101 98 98
query85 1393 370 365 365
query86 487 301 322 301
query87 4544 4341 4277 4277
query88 4583 2476 2480 2476
query89 441 318 321 318
query90 2118 225 233 225
query91 151 131 126 126
query92 85 78 75 75
query93 5158 550 551 550
query94 994 332 320 320
query95 397 298 333 298
query96 613 286 292 286
query97 3224 3132 3113 3113
query98 265 226 230 226
query99 1808 1334 1305 1305
Total cold run time: 327802 ms
Total hot run time: 196645 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2303703071
<details>
<summary>ClickBench: <b>Total hot run time: 30.69 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 740144c09664bd5fcc4a77551e5f4fdf659aa12d, data reload: false
query1 0.05 0.05 0.04
query2 0.08 0.04 0.04
query3 0.22 0.06 0.05
query4 1.67 0.10 0.09
query5 0.52 0.49 0.49
query6 1.15 0.74 0.72
query7 0.02 0.01 0.01
query8 0.06 0.05 0.05
query9 0.55 0.49 0.49
query10 0.55 0.55 0.55
query11 0.16 0.12 0.12
query12 0.16 0.12 0.14
query13 0.61 0.60 0.59
query14 0.77 0.80 0.79
query15 0.83 0.83 0.84
query16 0.37 0.37 0.38
query17 1.06 1.06 1.01
query18 0.22 0.21 0.20
query19 1.88 1.76 1.78
query20 0.02 0.01 0.01
query21 15.43 0.67 0.65
query22 4.20 7.79 1.49
query23 18.28 1.48 1.25
query24 2.07 0.27 0.22
query25 0.17 0.09 0.09
query26 0.27 0.18 0.19
query27 0.08 0.09 0.08
query28 13.23 1.03 1.01
query29 12.70 3.37 3.33
query30 0.43 0.25 0.24
query31 2.80 0.41 0.41
query32 3.23 0.49 0.49
query33 2.94 2.95 2.98
query34 16.96 4.36 4.40
query35 4.48 4.47 4.37
query36 0.67 0.49 0.51
query37 0.21 0.18 0.17
query38 0.17 0.16 0.17
query39 0.06 0.06 0.05
query40 0.19 0.16 0.16
query41 0.12 0.07 0.07
query42 0.07 0.07 0.08
query43 0.07 0.07 0.07
Total cold run time: 109.78 s
Total hot run time: 30.69 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1730411458
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1119,6 +1200,12 @@ int InstanceRecycler::recycle_tablets(int64_t table_id, int64_t index_id, int64_
return -1;
}
int64_t tablet_id = tablet_meta_pb.tablet_id();
+
+ if (!check_lazy_txn_finished(txn_kv_, instance_id_, tablet_meta_pb.tablet_id())) {
+ LOG(WARNING) << "lazy txn not finished tablet_id=" << tablet_meta_pb.tablet_id();
Review Comment:
LOG
"lazy txn not finished, skip recycling. tablet_id= ... txn_id ..."
and add a bvar to monitor `skip_recyle_tablet_lazy_txn`
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1222,6 +1309,10 @@ int InstanceRecycler::recycle_tablets(int64_t table_id, int64_t index_id, int64_
int ret = scan_and_recycle(tablet_key_begin, tablet_key_end, std::move(recycle_func),
std::move(loop_done));
+ if (ret != 0) {
+ LOG(WARNING) << "failed to scan_and_recycle, instance_id=" << instance_id_;
Review Comment:
also log table id index_id partition id and db id (if has)
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1707,6 +1798,64 @@ int InstanceRecycler::recycle_rowsets() {
return ret;
}
+bool check_txn_abort(std::shared_ptr<TxnKv> txn_kv, const std::string& instance_id,
Review Comment:
add comment for to describe the behavior
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -824,6 +831,78 @@ int InstanceRecycler::recycle_indexes() {
return scan_and_recycle(index_key0, index_key1, std::move(recycle_func), std::move(loop_done));
}
+bool check_lazy_txn_finished(std::shared_ptr<TxnKv> txn_kv, const std::string instance_id,
+ int64_t tablet_id) {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ LOG(WARNING) << "failed to create txn, instance_id=" << instance_id
+ << " tablet_id=" << tablet_id << " err=" << err;
+ return false;
+ }
+
+ std::string tablet_idx_key = meta_tablet_idx_key({instance_id, tablet_id});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ LOG(WARNING) << "failed to get tablet index, instance_id=" << instance_id
+ << " tablet_id=" << tablet_id << " err=" << err
+ << " key=" << hex(tablet_idx_key);
+ return false;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ LOG(WARNING) << "failed to parse tablet_idx_pb, instance_id=" << instance_id
+ << " tablet_id=" << tablet_id;
+ return false;
+ }
+
+ if (!tablet_idx_pb.has_db_id()) {
+ return true;
+ }
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id, tablet_idx_pb.db_id(), tablet_idx_pb.table_id(),
+ tablet_idx_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ return true;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ LOG(WARNING) << "failed to get partition version, instance_id=" << instance_id
+ << " db_id=" << tablet_idx_pb.db_id()
+ << " table_id=" << tablet_idx_pb.table_id()
+ << " partition_id=" << tablet_idx_pb.partition_id()
+ << " tablet_id=" << tablet_id << " key=" << hex(ver_key) << " err=" << err;
+ return false;
+ }
+
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ LOG(WARNING) << "failed to parse version_pb, instance_id=" << instance_id
+ << " db_id=" << tablet_idx_pb.db_id()
+ << " table_id=" << tablet_idx_pb.table_id()
+ << " partition_id=" << tablet_idx_pb.partition_id()
+ << " tablet_id=" << tablet_id << " key=" << hex(ver_key);
+ return false;
+ }
+
+ if (version_pb.has_txn_id()) {
+ LOG(WARNING) << "lazy txn not finished, instance_id=" << instance_id
Review Comment:
it seems to be an INFO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732234447
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
Review Comment:
in order to check `decode_key` bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2259512593
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2260228866
<details>
<summary>TPC-DS: <b>Total hot run time: 169394 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae48134efcfe7887d2b0b6f2d645e9c4fac57e59, data reload: false
query1 916 366 366 366
query2 6454 1687 1579 1579
query3 6665 210 226 210
query4 20233 17151 17102 17102
query5 3666 515 513 513
query6 274 164 165 164
query7 4602 307 291 291
query8 266 200 193 193
query9 8499 2403 2384 2384
query10 449 282 265 265
query11 10473 9989 9921 9921
query12 124 89 87 87
query13 1642 382 380 380
query14 10035 7426 7738 7426
query15 200 168 160 160
query16 6770 449 493 449
query17 962 568 559 559
query18 1824 286 290 286
query19 194 150 147 147
query20 93 83 82 82
query21 210 101 97 97
query22 4261 3995 3887 3887
query23 34028 33760 33757 33757
query24 10000 3134 3046 3046
query25 677 418 422 418
query26 1665 155 152 152
query27 2945 295 291 291
query28 7387 2035 2023 2023
query29 1128 457 410 410
query30 238 154 158 154
query31 952 769 770 769
query32 104 53 57 53
query33 711 324 339 324
query34 944 492 513 492
query35 885 753 733 733
query36 1026 894 876 876
query37 164 78 79 78
query38 2863 2756 2723 2723
query39 870 848 798 798
query40 252 118 112 112
query41 47 45 46 45
query42 124 97 106 97
query43 450 404 419 404
query44 1193 728 739 728
query45 222 175 174 174
query46 1086 827 789 789
query47 1798 1684 1713 1684
query48 361 301 297 297
query49 911 417 413 413
query50 893 439 436 436
query51 6735 6717 6663 6663
query52 104 91 101 91
query53 258 181 184 181
query54 641 444 446 444
query55 73 74 75 74
query56 279 259 250 250
query57 1138 1036 991 991
query58 293 259 269 259
query59 2535 2316 2304 2304
query60 291 288 302 288
query61 109 92 93 92
query62 889 652 641 641
query63 207 189 177 177
query64 5635 1892 1873 1873
query65 3148 3089 3069 3069
query66 1043 397 335 335
query67 15145 14843 14737 14737
query68 4287 564 577 564
query69 451 308 295 295
query70 1089 1034 1087 1034
query71 395 274 282 274
query72 7077 2697 2468 2468
query73 762 329 328 328
query74 5960 5652 5565 5565
query75 3355 2710 2740 2710
query76 2207 1355 1397 1355
query77 474 320 326 320
query78 9431 9207 8865 8865
query79 2129 528 544 528
query80 860 517 521 517
query81 542 225 225 225
query82 773 128 131 128
query83 189 170 177 170
query84 278 81 81 81
query85 1426 311 300 300
query86 456 316 312 312
query87 3217 3065 3099 3065
query88 3722 2429 2416 2416
query89 397 305 281 281
query90 1819 191 193 191
query91 126 102 101 101
query92 60 50 50 50
query93 1791 623 626 623
query94 758 279 295 279
query95 385 266 261 261
query96 603 284 281 281
query97 3191 3035 3061 3035
query98 218 242 197 197
query99 1693 1309 1282 1282
Total cold run time: 261380 ms
Total hot run time: 169394 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2260682341
<details>
<summary>TPC-H: <b>Total hot run time: 41756 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f23de131c772e5e5b480379dddd6cae001bf04a6, data reload: false
------ Round 1 ----------------------------------
q1 17683 4293 4089 4089
q2 2021 210 222 210
q3 10514 1328 1328 1328
q4 10222 849 994 849
q5 7669 2997 3009 2997
q6 221 134 134 134
q7 1032 607 608 607
q8 9438 1890 1953 1890
q9 8521 6640 6632 6632
q10 8723 3894 3832 3832
q11 429 247 255 247
q12 409 220 223 220
q13 17770 2941 2968 2941
q14 271 244 244 244
q15 529 481 498 481
q16 525 394 390 390
q17 985 951 920 920
q18 8124 7265 7217 7217
q19 1476 1231 1231 1231
q20 573 337 340 337
q21 5279 4676 4710 4676
q22 356 284 284 284
Total cold run time: 112770 ms
Total hot run time: 41756 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4093 4023 3985 3985
q2 324 222 219 219
q3 2995 2981 3226 2981
q4 2048 2025 1978 1978
q5 5676 5513 5466 5466
q6 218 132 152 132
q7 2148 1844 1821 1821
q8 3349 3425 3376 3376
q9 8737 8768 8802 8768
q10 3990 4100 3901 3901
q11 563 447 461 447
q12 779 622 650 622
q13 16510 3123 3121 3121
q14 317 275 270 270
q15 531 493 483 483
q16 471 430 422 422
q17 1777 1743 1749 1743
q18 8421 7802 7804 7802
q19 1724 1716 1781 1716
q20 2053 1878 1858 1858
q21 5787 5532 5303 5303
q22 543 489 455 455
Total cold run time: 73054 ms
Total hot run time: 56869 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732029358
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -265,6 +267,20 @@ void MetaServiceImpl::get_version(::google::protobuf::RpcController* controller,
msg = "malformed version value";
return;
}
+
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
Review Comment:
> how do we deal with the "duplicated" submmits to to the commiter add comment here.
txn_lazy_committer_ has a map indent by txn_id used for deduplicate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312460807
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312974858
<details>
<summary>ClickBench: <b>Total hot run time: 30.99 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40557e2131ead7df254d700743017f557f0777e6, data reload: false
query1 0.04 0.05 0.03
query2 0.08 0.04 0.04
query3 0.22 0.04 0.05
query4 1.67 0.07 0.08
query5 0.52 0.51 0.51
query6 1.12 0.74 0.73
query7 0.02 0.01 0.01
query8 0.05 0.04 0.04
query9 0.56 0.51 0.50
query10 0.54 0.54 0.55
query11 0.15 0.11 0.12
query12 0.15 0.13 0.13
query13 0.62 0.60 0.59
query14 0.74 0.82 0.78
query15 0.83 0.82 0.82
query16 0.38 0.38 0.38
query17 1.07 1.06 1.04
query18 0.20 0.20 0.20
query19 1.95 1.85 1.89
query20 0.01 0.02 0.01
query21 15.41 0.65 0.65
query22 3.95 8.06 2.03
query23 18.28 1.33 1.28
query24 2.16 0.20 0.21
query25 0.16 0.08 0.07
query26 0.26 0.17 0.18
query27 0.07 0.07 0.07
query28 13.22 1.01 0.99
query29 12.63 3.37 3.37
query30 0.24 0.06 0.06
query31 2.89 0.39 0.39
query32 3.28 0.48 0.48
query33 3.01 3.02 3.01
query34 17.00 4.34 4.38
query35 4.45 4.41 4.38
query36 0.65 0.51 0.48
query37 0.20 0.15 0.15
query38 0.15 0.15 0.14
query39 0.04 0.04 0.04
query40 0.15 0.14 0.14
query41 0.10 0.04 0.04
query42 0.07 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 109.34 s
Total hot run time: 30.99 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732710627
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -862,94 +863,485 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
std::stringstream ss;
int64_t txn_id = request->txn_id();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // Get txn info with db_id and txn_id
- std::string info_val; // Will be reused when saving updated txn
- const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
- err = txn->get(info_key, &info_val);
- if (err != TxnErrorCode::TXN_OK) {
- code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
- : cast_as<ErrCategory::READ>(err);
- if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
- } else {
- ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
- << " err=" << err;
+ // Get txn info with db_id and txn_id
+ std::string info_val; // Will be reused when saving updated txn
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
+ } else {
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
+ }
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(info_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // TODO: do more check like txn state, 2PC etc.
- DCHECK(txn_info.txn_id() == txn_id);
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
- code = MetaServiceCode::TXN_ALREADY_ABORTED;
- ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ // TODO: do more check like txn state, 2PC etc.
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
- if (request->has_is_2pc() && request->is_2pc()) {
- code = MetaServiceCode::TXN_ALREADY_VISIBLE;
- ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(INFO) << msg;
response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- code = MetaServiceCode::OK;
- ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(INFO) << msg;
- response->mutable_txn_info()->CopyFrom(txn_info);
- return;
- }
- if (request->has_is_2pc() && request->is_2pc() &&
- txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
- code = MetaServiceCode::TXN_INVALID_STATUS;
- ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Prepare rowset meta and new_versions
+ // Prepare rowset meta and new_versions
+ // Read tablet indexes in batch.
+ std::vector<std::string> tablet_idx_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
+ }
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ size_t total_rowsets = tmp_rowsets_meta.size();
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ for (size_t i = 0; i < total_rowsets; i++) {
+ uint64_t tablet_id = tmp_rowsets_meta[i].second.tablet_id();
+ if (!tablet_idx_values[i].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " tablet_id=" << tablet_id << " key=" << hex(tablet_idx_keys[i]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "malformed tablet index value tablet_id=" << tablet_id
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ VLOG_DEBUG << "tablet_id:" << tablet_id
+ << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ }
+
+ tablet_idx_keys.clear();
+ tablet_idx_values.clear();
+
+ // {table/partition} -> version
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ size_t total_versions = version_keys.size();
+ for (size_t i = 0; i < total_versions; i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312652072
<details>
<summary>TPC-H: <b>Total hot run time: 38057 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5f3b5ea8bfb6214887e8d1bb42d8f87233ccc8a0, data reload: false
------ Round 1 ----------------------------------
q1 18300 4642 4298 4298
q2 2049 190 176 176
q3 11534 953 1155 953
q4 10498 741 668 668
q5 7772 2912 2821 2821
q6 225 137 140 137
q7 969 630 601 601
q8 9325 2067 2122 2067
q9 7211 6523 6553 6523
q10 7002 2171 2180 2171
q11 452 251 249 249
q12 403 234 232 232
q13 18836 3080 3030 3030
q14 276 238 243 238
q15 518 480 499 480
q16 511 390 391 390
q17 999 715 757 715
q18 7423 7041 6928 6928
q19 1393 1055 989 989
q20 691 342 335 335
q21 4141 3044 3145 3044
q22 1087 1033 1012 1012
Total cold run time: 111615 ms
Total hot run time: 38057 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4327 4278 4279 4278
q2 377 281 278 278
q3 2901 2677 2683 2677
q4 1914 1695 1680 1680
q5 5700 5723 5833 5723
q6 226 135 138 135
q7 2210 1888 1902 1888
q8 3298 3454 3544 3454
q9 8868 8868 8851 8851
q10 3656 3418 3367 3367
q11 625 519 505 505
q12 834 705 670 670
q13 14293 3137 3267 3137
q14 334 281 283 281
q15 556 518 499 499
q16 493 445 442 442
q17 1856 1557 1570 1557
q18 8302 7890 7960 7890
q19 1763 1538 1721 1538
q20 2181 1911 1922 1911
q21 5724 5504 5413 5413
q22 1183 1033 1031 1031
Total cold run time: 71621 ms
Total hot run time: 57205 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2244960031
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1692571333
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -237,46 +238,65 @@ void MetaServiceImpl::get_version(::google::protobuf::RpcController* controller,
partition_version_key({instance_id, db_id, table_id, partition_id}, &ver_key);
}
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv_->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- msg = "failed to create txn";
- code = cast_as<ErrCategory::CREATE>(err);
- return;
- }
+ do {
+ code = MetaServiceCode::OK;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ msg = "failed to create txn";
+ code = cast_as<ErrCategory::CREATE>(err);
+ return;
+ }
- std::string ver_val;
- // 0 for success get a key, 1 for key not found, negative for error
- err = txn->get(ver_key, &ver_val);
- VLOG_DEBUG << "xxx get version_key=" << hex(ver_key);
- if (err == TxnErrorCode::TXN_OK) {
- if (is_table_version) {
- int64_t version = 0;
- if (!txn->decode_atomic_int(ver_val, &version)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "malformed table version value";
- return;
- }
- response->set_version(version);
- } else {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(ver_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "malformed version value";
- return;
+ std::string ver_val;
+ // 0 for success get a key, 1 for key not found, negative for error
+ err = txn->get(ver_key, &ver_val);
+ VLOG_DEBUG << "xxx get version_key=" << hex(ver_key);
+ if (err == TxnErrorCode::TXN_OK) {
+ if (is_table_version) {
+ int64_t version = 0;
+ if (!txn->decode_atomic_int(ver_val, &version)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "malformed table version value";
+ return;
+ }
+ response->set_version(version);
+ } else {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "malformed version value";
+ return;
+ }
+
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ code = ret.first;
+ msg = ret.second;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING)
+ << "wait txn lazy commit failed, txn_id=" << version_pb.txn_id();
+ return;
+ }
+ continue;
Review Comment:
This can be simplified a bit. After the `wait()` ends, the transaction has already been committed, and the corresponding version is visible. We can return this version directly without retrying.
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -387,10 +412,30 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
msg = "malformed version value";
break;
}
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ code = ret.first;
+ msg = ret.second;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "wait txn lazy commit failed, txn_id="
+ << version_pb.txn_id();
+ break;
+ }
+ goto TRY_AGAIN;
Review Comment:
Directly fetching all versions and then waiting for all transactions at once is a simpler approach. For now, let's keep the current method to push the PR merge, and we can address this improvement in another PR.
##########
cloud/src/meta-service/txn_lazy_committer.h:
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/cloud.pb.h>
+
+#include <atomic>
+
+#include "common/simple_thread_pool.h"
+#include "meta-service/txn_kv.h"
+
+namespace doris::cloud {
+
+class TxnLazyCommitter;
+
+class TxnLazyCommitTask {
+public:
+ TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer_);
Review Comment:
```suggestion
TxnLazyCommitter* txn_lazy_committer);
```
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+extern void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+extern void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+extern void make_committed_txn_visible(const std::string& instance_id, int64_t db_id,
+ int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg);
Review Comment:
`extern` is not required here.
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+extern void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+extern void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+extern void make_committed_txn_visible(const std::string& instance_id, int64_t db_id,
+ int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id,
+ int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+ DCHECK(txn_lazy_committer != nullptr);
+ txn_id_ = txn_id;
+ txn_lazy_committer_ = txn_lazy_committer;
Review Comment:
Initialize in member initializer list, likes:
```c++
TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
std::shared_ptr<TxnKv> txn_kv, TxnLazyCommitter* txn_lazy_committer)
: instance_id_(instance_id),
txn_id_(txn_id),
txn_kv_(std::move(txn_id)),
txn_lazy_committer_(txn_lazy_committer) {
DCHECK(txn_id > 0);
// ...
}
```
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+extern void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+extern void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+extern void make_committed_txn_visible(const std::string& instance_id, int64_t db_id,
+ int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id,
+ int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+ DCHECK(txn_lazy_committer != nullptr);
+ txn_id_ = txn_id;
+ txn_lazy_committer_ = txn_lazy_committer;
+ task_func_ = [=, this]() {
+ MetaServiceCode code;
+ std::stringstream ss;
+ std::string msg;
+
+ do {
+ code = MetaServiceCode::OK;
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id, txn_id, txn_kv, code, msg, &db_id, &tmp_rowset_metas);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id << " code=" << code;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id;
+ break;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end =
+ (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id, txn_id, txn_kv, code, msg, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code != MetaServiceCode::OK) break;
+ }
+ if (code != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND
+ ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ break;
+ }
+ }
+ make_committed_txn_visible(instance_id, db_id, txn_id, txn_kv, code, msg);
+ } while (false);
+
+ std::unique_lock<std::mutex> lock(mutex_);
+ this->code_ = code;
+ this->msg_ = std::move(msg);
+ this->finished_ = true;
+ this->cond_.notify_all();
+ };
+}
+
+std::pair<MetaServiceCode, std::string> TxnLazyCommitTask::wait() {
+ std::unique_lock<std::mutex> lock(mutex_);
+ cond_.wait(lock, [this]() { return this->finished_ == true; });
+ txn_lazy_committer_->remove(txn_id_);
+ return std::make_pair(this->code_, this->msg_);
+}
+
+TxnLazyCommitter::TxnLazyCommitter(std::shared_ptr<TxnKv> txn_kv) {
+ txn_kv_ = txn_kv;
+ worker_pool_ = std::make_unique<SimpleThreadPool>(config::txn_lazy_commit_worker_num);
+ worker_pool_->start();
+}
+
+std::shared_ptr<TxnLazyCommitTask> TxnLazyCommitter::submit(const std::string& instance_id,
+ int64_t txn_id) {
+ std::unique_lock<std::mutex> lock(mutex_);
+ auto iter = running_tasks_.find(txn_id);
+ if (iter != running_tasks_.end()) {
+ return iter->second;
+ }
+
+ auto task = std::make_shared<TxnLazyCommitTask>(instance_id, txn_id, txn_kv_, this);
+ running_tasks_.emplace(txn_id, task);
+ worker_pool_->submit(task->task_func_);
+ return task;
Review Comment:
The use of locks here can be further optimized. For example, it can be modified as follows:
```suggestion
std::shared_ptr<TxnLazyCommitTask> task;
{
std::unique_lock<std::mutex> lock(mutex_);
auto iter = running_tasks_.find(txn_id);
if (iter != running_tasks_.end()) {
return iter->second;
}
task = std::make_shared<TxnLazyCommitTask>(instance_id, txn_id, txn_kv_, this);
running_tasks_.emplace(txn_id, task);
}
worker_pool_->submit(task->task_func_);
return task;
```
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+extern void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+extern void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+extern void make_committed_txn_visible(const std::string& instance_id, int64_t db_id,
+ int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id,
+ int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+ DCHECK(txn_lazy_committer != nullptr);
+ txn_id_ = txn_id;
+ txn_lazy_committer_ = txn_lazy_committer;
+ task_func_ = [=, this]() {
+ MetaServiceCode code;
+ std::stringstream ss;
+ std::string msg;
+
+ do {
+ code = MetaServiceCode::OK;
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id, txn_id, txn_kv, code, msg, &db_id, &tmp_rowset_metas);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id << " code=" << code;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id;
+ break;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
Review Comment:
```suggestion
for (auto& rowset_pair : tmp_rowset_metas) {
int64_t partition_id = rowset_pair.second.partition_id();
partition_to_tmp_rowset_metas[partition_id].push_back(std::move(rowset_pair));
```
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -788,6 +788,164 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
txn->put(key, val);
}
}
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (tablet_ids.find(tmp_rowset_pb.tablet_id()) == tablet_ids.end()) {
+ std::string tablet_idx_key;
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()}, &tablet_idx_key);
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (partition_versions.find(tmp_rowset_pb.partition_id()) == partition_versions.end()) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
Review Comment:
There may be conflicts caused by concurrent commits. According to our design expectations, TXN_CONFLICT also needs to be handled. One simple solution is to consider the data already converted if TXN_CONFLICT occurs (this requires that each task uses the same split batch method). Another solution is to retry once if TXN_CONFLICT occurs.
It's essential to manage the competition between transactions properly here.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -913,320 +1458,420 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
- msg = ss.str();
- return;
- }
- version = version_pb.version();
- } else {
- version = 1;
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
}
- new_versions[version_keys[i]] = version + 1;
+ txn->put(recycle_key, recycle_val);
}
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
Review Comment:
Handle `TXN_CONFLICT`.
##########
cloud/src/meta-service/txn_lazy_committer.h:
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/cloud.pb.h>
+
+#include <atomic>
+
+#include "common/simple_thread_pool.h"
+#include "meta-service/txn_kv.h"
+
+namespace doris::cloud {
+
+class TxnLazyCommitter;
+
+class TxnLazyCommitTask {
+public:
+ TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer_);
+
+ std::pair<MetaServiceCode, std::string> wait();
Review Comment:
How about adding a comment?
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -804,91 +962,483 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
Review Comment:
`std::shared_ptr` inherently carries ownership semantics. If `commit_txn_immediately` does not need to transfer the `shared_ptr` (eg, inc reference count), then using a reference would be more appropriate (making the semantics more accurate). For example, `shared_ptr<TxnKv>&` can be changed to `TxnKv&`.
##########
cloud/src/meta-service/txn_lazy_committer.h:
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/cloud.pb.h>
+
+#include <atomic>
+
+#include "common/simple_thread_pool.h"
+#include "meta-service/txn_kv.h"
+
+namespace doris::cloud {
+
+class TxnLazyCommitter;
+
+class TxnLazyCommitTask {
+public:
+ TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer_);
+
+ std::pair<MetaServiceCode, std::string> wait();
+
+private:
+ friend class TxnLazyCommitter;
+ int64_t txn_id_;
Review Comment:
```suggestion
friend class TxnLazyCommitter;
int64_t txn_id_;
```
##########
cloud/src/meta-service/txn_lazy_committer.h:
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/cloud.pb.h>
+
+#include <atomic>
+
+#include "common/simple_thread_pool.h"
+#include "meta-service/txn_kv.h"
+
+namespace doris::cloud {
+
+class TxnLazyCommitter;
+
+class TxnLazyCommitTask {
Review Comment:
`TxnLazyCommitTask` doesn't need to be exposed to external users, so it can be defined in the cpp file to speed up compilation. This will introduce the issue of `TxnLazyCommitter` not finding the `TxnLazyCommitTask` definition during destruction, which can be resolved by adding `TxnLazyCommitter::~TxnLazyCommitter() = default;` in the cpp file.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -788,6 +788,164 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
txn->put(key, val);
}
}
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
Review Comment:
Add a debug log.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -788,6 +788,164 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
txn->put(key, val);
}
}
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (tablet_ids.find(tmp_rowset_pb.tablet_id()) == tablet_ids.end()) {
+ std::string tablet_idx_key;
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()}, &tablet_idx_key);
Review Comment:
```suggestion
std::string tablet_idx_key = meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
```
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+extern void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+extern void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+extern void make_committed_txn_visible(const std::string& instance_id, int64_t db_id,
+ int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id,
+ int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+ DCHECK(txn_lazy_committer != nullptr);
+ txn_id_ = txn_id;
+ txn_lazy_committer_ = txn_lazy_committer;
+ task_func_ = [=, this]() {
+ MetaServiceCode code;
+ std::stringstream ss;
+ std::string msg;
+
+ do {
+ code = MetaServiceCode::OK;
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id, txn_id, txn_kv, code, msg, &db_id, &tmp_rowset_metas);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id << " code=" << code;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id;
+ break;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end =
+ (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id, txn_id, txn_kv, code, msg, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code != MetaServiceCode::OK) break;
+ }
+ if (code != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND
+ ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ break;
+ }
+ }
+ make_committed_txn_visible(instance_id, db_id, txn_id, txn_kv, code, msg);
+ } while (false);
+
+ std::unique_lock<std::mutex> lock(mutex_);
+ this->code_ = code;
+ this->msg_ = std::move(msg);
+ this->finished_ = true;
+ this->cond_.notify_all();
+ };
+}
+
+std::pair<MetaServiceCode, std::string> TxnLazyCommitTask::wait() {
+ std::unique_lock<std::mutex> lock(mutex_);
+ cond_.wait(lock, [this]() { return this->finished_ == true; });
+ txn_lazy_committer_->remove(txn_id_);
+ return std::make_pair(this->code_, this->msg_);
+}
+
+TxnLazyCommitter::TxnLazyCommitter(std::shared_ptr<TxnKv> txn_kv) {
Review Comment:
Initialize in the member initializer list too
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+extern void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+extern void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+extern void make_committed_txn_visible(const std::string& instance_id, int64_t db_id,
+ int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id,
+ int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+ DCHECK(txn_lazy_committer != nullptr);
+ txn_id_ = txn_id;
+ txn_lazy_committer_ = txn_lazy_committer;
+ task_func_ = [=, this]() {
Review Comment:
Declare this as a member function of `TxnLazyCommitTask`, eg: `TxnLazyCommiter::commit` then change `TxnLazyCommitter::submit` to:
```c++
auto task = std::make_shared<TxnLazyCommitTask>(instance_id, txn_id, txn_kv_, this);
running_tasks_.emplace(txn_id, task);
worker_pool_->submit([task]() { task->commit(); });
```
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -788,6 +788,164 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
txn->put(key, val);
}
}
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (tablet_ids.find(tmp_rowset_pb.tablet_id()) == tablet_ids.end()) {
Review Comment:
```suggestion
if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
```
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -1810,8 +2455,17 @@ void MetaServiceImpl::commit_txn(::google::protobuf::RpcController* controller,
LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id << " code=" << code;
return;
}
- commit_txn_immediately(request, response, txn_kv_, code, msg, instance_id, db_id,
- tmp_rowsets_meta);
+ VLOG_DEBUG << "txn_id=" << txn_id << " tmp_rowset size=" << tmp_rowsets_meta.size();
+ if (config::enable_txn_lazy_commit &&
+ tmp_rowsets_meta.size() >= config::txn_lazy_commit_rowsets_thresold) {
+ commit_txn_eventually(request, response, txn_kv_, txn_lazy_committer_, code, msg,
+ instance_id, db_id, tmp_rowsets_meta);
+ return;
+ }
+
+ commit_txn_immediately(request, response, txn_kv_, txn_lazy_committer_, code, msg, instance_id,
+ db_id, tmp_rowsets_meta);
+ return;
Review Comment:
Redundant `return`.
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+extern void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+extern void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+extern void make_committed_txn_visible(const std::string& instance_id, int64_t db_id,
+ int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id,
+ int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+ DCHECK(txn_lazy_committer != nullptr);
+ txn_id_ = txn_id;
+ txn_lazy_committer_ = txn_lazy_committer;
+ task_func_ = [=, this]() {
+ MetaServiceCode code;
+ std::stringstream ss;
+ std::string msg;
+
+ do {
+ code = MetaServiceCode::OK;
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id, txn_id, txn_kv, code, msg, &db_id, &tmp_rowset_metas);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id << " code=" << code;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id;
+ break;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end =
+ (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id, txn_id, txn_kv, code, msg, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code != MetaServiceCode::OK) break;
+ }
+ if (code != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND
+ ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
Review Comment:
Handle `TXN_CONFLICT` too.
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+extern void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+extern void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+extern void make_committed_txn_visible(const std::string& instance_id, int64_t db_id,
+ int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id,
+ int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+ DCHECK(txn_lazy_committer != nullptr);
+ txn_id_ = txn_id;
+ txn_lazy_committer_ = txn_lazy_committer;
+ task_func_ = [=, this]() {
+ MetaServiceCode code;
+ std::stringstream ss;
+ std::string msg;
+
+ do {
+ code = MetaServiceCode::OK;
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id, txn_id, txn_kv, code, msg, &db_id, &tmp_rowset_metas);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id << " code=" << code;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id;
+ break;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end =
+ (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id, txn_id, txn_kv, code, msg, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code != MetaServiceCode::OK) break;
+ }
+ if (code != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND
+ ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ break;
+ }
+ }
+ make_committed_txn_visible(instance_id, db_id, txn_id, txn_kv, code, msg);
+ } while (false);
+
+ std::unique_lock<std::mutex> lock(mutex_);
+ this->code_ = code;
+ this->msg_ = std::move(msg);
+ this->finished_ = true;
+ this->cond_.notify_all();
+ };
+}
+
+std::pair<MetaServiceCode, std::string> TxnLazyCommitTask::wait() {
+ std::unique_lock<std::mutex> lock(mutex_);
+ cond_.wait(lock, [this]() { return this->finished_ == true; });
+ txn_lazy_committer_->remove(txn_id_);
+ return std::make_pair(this->code_, this->msg_);
Review Comment:
`txn_lazy_committer_->remove(txn_id_)` has lock, try:
```suggestion
{
std::unique_lock<std::mutex> lock(mutex_);
cond_.wait(lock, [this]() { return this->finished_ == true; });
}
txn_lazy_committer_->remove(txn_id_);
// The lock has memory barriers, so access code_, msg_ without lock is safety.
return std::make_pair(this->code_, this->msg_);
```
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -788,6 +788,164 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
txn->put(key, val);
}
}
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
Review Comment:
`partition_versions` is not needed here, since all rowsets belong to the same partition. Read the partition version in here directly.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -788,6 +788,164 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
txn->put(key, val);
}
}
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (tablet_ids.find(tmp_rowset_pb.tablet_id()) == tablet_ids.end()) {
+ std::string tablet_idx_key;
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()}, &tablet_idx_key);
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val);
Review Comment:
Would using snapshot semantics be sufficient?
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -788,6 +788,164 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
txn->put(key, val);
}
}
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (tablet_ids.find(tmp_rowset_pb.tablet_id()) == tablet_ids.end()) {
+ std::string tablet_idx_key;
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()}, &tablet_idx_key);
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (partition_versions.find(tmp_rowset_pb.partition_id()) == partition_versions.end()) {
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2249608295
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2244670548
<details>
<summary>ClickBench: <b>Total hot run time: 30.71 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0cac9c57dfbabd3aa3c3d90819462ad7e839f881, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.05
query3 0.22 0.05 0.05
query4 1.68 0.07 0.07
query5 0.51 0.48 0.50
query6 1.14 0.72 0.72
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.56 0.50 0.49
query10 0.54 0.55 0.55
query11 0.16 0.10 0.12
query12 0.14 0.12 0.12
query13 0.59 0.58 0.58
query14 0.76 0.78 0.81
query15 0.85 0.82 0.83
query16 0.36 0.36 0.37
query17 0.99 1.02 1.03
query18 0.23 0.21 0.22
query19 1.93 1.84 1.74
query20 0.01 0.01 0.01
query21 15.39 0.77 0.68
query22 4.09 7.48 1.93
query23 18.33 1.33 1.29
query24 2.10 0.23 0.22
query25 0.14 0.09 0.08
query26 0.31 0.22 0.20
query27 0.46 0.23 0.23
query28 13.26 1.02 0.99
query29 12.61 3.32 3.29
query30 0.25 0.06 0.05
query31 2.88 0.40 0.39
query32 3.26 0.49 0.46
query33 2.87 2.89 2.88
query34 17.03 4.37 4.36
query35 4.44 4.43 4.41
query36 0.65 0.46 0.48
query37 0.19 0.15 0.16
query38 0.15 0.14 0.16
query39 0.04 0.04 0.03
query40 0.16 0.12 0.13
query41 0.10 0.04 0.06
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 109.68 s
Total hot run time: 30.71 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2244499494
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2298409527
<details>
<summary>ClickBench: <b>Total hot run time: 30.87 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1d63b327147196a6029b3421305f1eed1a00dc18, data reload: false
query1 0.05 0.04 0.04
query2 0.08 0.04 0.04
query3 0.23 0.05 0.06
query4 1.67 0.07 0.08
query5 0.51 0.49 0.50
query6 1.14 0.74 0.73
query7 0.02 0.02 0.02
query8 0.06 0.05 0.06
query9 0.54 0.49 0.49
query10 0.56 0.56 0.55
query11 0.17 0.12 0.12
query12 0.15 0.13 0.13
query13 0.61 0.60 0.60
query14 0.76 0.77 0.80
query15 0.86 0.83 0.83
query16 0.38 0.38 0.38
query17 1.05 1.06 1.05
query18 0.22 0.20 0.21
query19 1.80 1.77 1.84
query20 0.01 0.01 0.01
query21 15.40 0.67 0.66
query22 4.44 7.97 1.51
query23 18.28 1.34 1.30
query24 2.12 0.23 0.23
query25 0.16 0.09 0.09
query26 0.28 0.19 0.19
query27 0.09 0.08 0.08
query28 13.27 1.03 1.02
query29 12.66 3.36 3.30
query30 0.43 0.24 0.24
query31 2.91 0.41 0.40
query32 3.80 0.49 0.50
query33 2.98 2.94 3.00
query34 17.01 4.37 4.41
query35 4.37 4.49 4.44
query36 0.67 0.49 0.51
query37 0.21 0.18 0.17
query38 0.17 0.16 0.18
query39 0.07 0.05 0.06
query40 0.18 0.15 0.16
query41 0.12 0.07 0.07
query42 0.08 0.08 0.07
query43 0.06 0.06 0.06
Total cold run time: 110.63 s
Total hot run time: 30.87 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2302092292
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2301232335
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2306018891
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2283201473
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2285289203
<details>
<summary>TPC-H: <b>Total hot run time: 40582 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e1cd3f4d183656b41f4319a5380401fc4759cff, data reload: false
------ Round 1 ----------------------------------
q1 17610 4422 4349 4349
q2 2034 189 184 184
q3 11983 1020 1100 1020
q4 10542 761 765 761
q5 7787 2819 2869 2819
q6 227 142 142 142
q7 981 599 594 594
q8 9389 2088 2095 2088
q9 8830 6615 6629 6615
q10 7040 2196 2193 2193
q11 482 252 257 252
q12 401 232 229 229
q13 17763 3021 2992 2992
q14 274 243 240 240
q15 526 470 495 470
q16 635 523 516 516
q17 1013 680 685 680
q18 8213 7387 7555 7387
q19 1388 1029 983 983
q20 794 423 419 419
q21 5456 4655 4665 4655
q22 1156 1011 994 994
Total cold run time: 114524 ms
Total hot run time: 40582 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4404 4342 4356 4342
q2 394 279 283 279
q3 2901 2698 2801 2698
q4 2055 1756 1744 1744
q5 5750 5648 5629 5629
q6 229 140 147 140
q7 2102 1779 1829 1779
q8 3368 3564 3471 3471
q9 8883 8882 8921 8882
q10 3583 3341 3284 3284
q11 595 512 531 512
q12 814 655 643 643
q13 15902 3237 3198 3198
q14 321 285 301 285
q15 582 509 491 491
q16 643 582 586 582
q17 1867 1572 1534 1534
q18 8180 8019 7829 7829
q19 6785 1586 1595 1586
q20 2285 1980 1944 1944
q21 5681 5524 5067 5067
q22 1175 1101 1083 1083
Total cold run time: 78499 ms
Total hot run time: 57002 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2291220320
<details>
<summary>TPC-H: <b>Total hot run time: 38245 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 758fe123926a613e6c373d8d814829c22a99a16a, data reload: false
------ Round 1 ----------------------------------
q1 19093 4869 4309 4309
q2 2073 213 223 213
q3 11670 1107 1083 1083
q4 10502 752 715 715
q5 7814 2882 2860 2860
q6 271 153 154 153
q7 997 644 641 641
q8 9374 2085 2106 2085
q9 7267 6568 6584 6568
q10 7081 2267 2194 2194
q11 485 266 262 262
q12 418 245 243 243
q13 17787 3029 3019 3019
q14 293 257 257 257
q15 557 524 520 520
q16 514 405 411 405
q17 987 708 713 708
q18 7496 6797 7037 6797
q19 5723 1084 1040 1040
q20 745 362 366 362
q21 4252 2770 3118 2770
q22 1138 1068 1041 1041
Total cold run time: 116537 ms
Total hot run time: 38245 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4486 4333 4263 4263
q2 415 306 307 306
q3 2853 2659 2644 2644
q4 1955 1702 1708 1702
q5 5718 5754 5708 5708
q6 244 152 161 152
q7 2225 1794 1759 1759
q8 3335 3555 3494 3494
q9 8870 8874 8770 8770
q10 3580 3412 3350 3350
q11 626 571 532 532
q12 830 663 642 642
q13 16047 3199 3212 3199
q14 337 300 287 287
q15 558 507 503 503
q16 495 456 446 446
q17 1865 1575 1566 1566
q18 8247 7913 7701 7701
q19 7784 1675 1546 1546
q20 2158 1876 1907 1876
q21 12767 5356 5371 5356
q22 1216 1080 1093 1080
Total cold run time: 86611 ms
Total hot run time: 56882 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1730627524
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
- // Save table versions
- for (auto& i : table_id_tablet_ids) {
- std::string ver_key = table_version_key({instance_id, db_id, i.first});
- txn->atomic_add(ver_key, 1);
- LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key) << " txn_id=" << txn_id;
- }
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << "txn_id=" << txn_id << " table_id=" << table_id
+ << " partition_id=" << partition_id << " version=" << i.second;
- LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
- // Update txn_info
- txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- auto now_time = system_clock::now();
- uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
- if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
- LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
- << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
- return;
- }
- txn_info.set_commit_time(commit_time);
- txn_info.set_finish_time(commit_time);
- if (request->has_commit_attachment()) {
- txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
- }
- LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
- info_val.clear();
- if (!txn_info.SerializeToString(&info_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(info_key, info_val);
- LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+ for (size_t i = 0; i < lock_keys.size(); i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- // Update stats of affected tablet
- for (auto& [tablet_id, stats] : tablet_stats) {
- DCHECK(tablet_ids.count(tablet_id));
- auto& tablet_idx = tablet_ids[tablet_id];
- StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
- tablet_idx.partition_id(), tablet_id};
- update_tablet_stats(info, stats, txn, code, msg);
- if (code != MetaServiceCode::OK) return;
- }
- // Remove tmp rowset meta
- for (auto& [k, _] : tmp_rowsets_meta) {
- txn->remove(k);
- LOG(INFO) << "xxx remove tmp_rowset_key=" << hex(k) << " txn_id=" << txn_id;
- }
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
- const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
- LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
- txn->remove(running_key);
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
- std::string recycle_val;
- std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
- RecycleTxnPB recycle_pb;
- recycle_pb.set_creation_time(commit_time);
- recycle_pb.set_label(txn_info.label());
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
+ << " txn_id=" << txn_id;
+ }
- if (!recycle_pb.SerializeToString(&recycle_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(recycle_key, recycle_val);
+ LOG(INFO) << "commit_txn_eventually put_size=" << txn->put_bytes()
+ << " del_size=" << txn->delete_bytes() << " num_put_keys=" << txn->num_put_keys()
+ << " num_del_keys=" << txn->num_del_keys()
+ << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
- if (txn_info.load_job_source_type() ==
- LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
- put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
- }
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
- LOG(INFO) << "xxx commit_txn put recycle_key key=" << hex(recycle_key) << " txn_id=" << txn_id;
- LOG(INFO) << "commit_txn put_size=" << txn->put_bytes() << " del_size=" << txn->delete_bytes()
- << " num_put_keys=" << txn->num_put_keys() << " num_del_keys=" << txn->num_del_keys()
- << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer->submit(instance_id, txn_id);
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ if (ret.first != MetaServiceCode::OK) {
+ LOG(WARNING) << "lazy commit txn failed txn_id=" << txn_id << " code=" << ret.first
+ << "msg=" << ret.second;
+ }
- // Finally we are done...
- err = txn->commit();
- if (err != TxnErrorCode::TXN_OK) {
- if ((err == TxnErrorCode::TXN_VALUE_TOO_LARGE ||
- err == TxnErrorCode::TXN_BYTES_TOO_LARGE) &&
- !tmp_rowsets_meta.empty()) {
- size_t max_size = 0, max_idx = 0, max_num_segments = 0,
- min_num_segments = std::numeric_limits<size_t>::max(), avg_num_segments = 0;
- for (size_t i = 0; i < tmp_rowsets_meta.size(); i++) {
- auto& [k, v] = tmp_rowsets_meta[i];
- if (v.ByteSizeLong() > max_size) {
- max_size = v.ByteSizeLong();
- max_idx = i;
- }
- if (v.num_segments() > max_num_segments) {
- max_num_segments = v.num_segments();
- }
- if (v.num_segments() < min_num_segments) {
- min_num_segments = v.num_segments();
- }
- avg_num_segments += v.num_segments();
- }
- if (!tmp_rowsets_meta.empty()) {
- avg_num_segments /= tmp_rowsets_meta.size();
- }
- LOG(WARNING) << "failed to commit kv txn"
- << ", err=" << err << ", txn_id=" << txn_id
- << ", total_rowsets=" << tmp_rowsets_meta.size()
- << ", avg_num_segments=" << avg_num_segments
- << ", min_num_segments=" << min_num_segments
- << ", max_num_segments=" << max_num_segments
- << ", largest_rowset_size=" << max_size
- << ", largest_rowset_key=" << hex(tmp_rowsets_meta[max_idx].first)
- << ", largest_rowset_value="
- << tmp_rowsets_meta[max_idx].second.ShortDebugString();
+ std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ // Accumulate affected rows
+ auto& stats = tablet_stats[i.tablet_id()];
+ stats.data_size += i.data_disk_size();
+ stats.num_rows += i.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += i.num_segments();
}
- code = cast_as<ErrCategory::COMMIT>(err);
- ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- return;
- }
- // calculate table stats from tablets stats
- std::map<int64_t /*table_id*/, TableStats> table_stats;
- std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
- request->base_tablet_ids().end());
- calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
- for (const auto& pair : table_stats) {
- TableStatsPB* stats_pb = response->add_table_stats();
- auto table_id = pair.first;
- auto stats = pair.second;
- get_pb_from_tablestats(stats, stats_pb);
- stats_pb->set_table_id(table_id);
- VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
- << " table_id=" << table_id
- << " updated_row_count=" << stats_pb->updated_row_count();
- }
+ // calculate table stats from tablets stats
+ std::map<int64_t /*table_id*/, TableStats> table_stats;
+ std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
+ request->base_tablet_ids().end());
+ calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
+ for (const auto& pair : table_stats) {
+ TableStatsPB* stats_pb = response->add_table_stats();
+ auto table_id = pair.first;
+ auto stats = pair.second;
+ get_pb_from_tablestats(stats, stats_pb);
+ stats_pb->set_table_id(table_id);
+ VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
+ << " table_id=" << table_id
+ << " updated_row_count=" << stats_pb->updated_row_count();
+ }
- response->mutable_txn_info()->CopyFrom(txn_info);
-} // end commit_txn
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
Review Comment:
add comment to describe why it is OK to set VISIBLE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1730953469
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
Review Comment:
> is it a unrecoverable error?
后续展开处理错误
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2275256551
<details>
<summary>TPC-H: <b>Total hot run time: 39663 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 53d56cfa3a39f18ab858f88c2903d1b32d180eff, data reload: false
------ Round 1 ----------------------------------
q1 17820 4496 4354 4354
q2 2663 182 179 179
q3 12196 1158 1105 1105
q4 10313 778 730 730
q5 7578 2503 2555 2503
q6 226 141 137 137
q7 975 602 589 589
q8 9226 1910 1885 1885
q9 8586 6517 6522 6517
q10 7056 2211 2143 2143
q11 466 245 250 245
q12 393 220 220 220
q13 17779 2943 2986 2943
q14 284 234 229 229
q15 525 496 484 484
q16 652 511 528 511
q17 973 617 706 617
q18 7994 7586 7459 7459
q19 5031 1004 1058 1004
q20 738 407 420 407
q21 5230 4438 4567 4438
q22 1064 964 1004 964
Total cold run time: 117768 ms
Total hot run time: 39663 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4520 4253 4273 4253
q2 382 279 275 275
q3 2797 2644 2588 2588
q4 1886 1635 1655 1635
q5 5297 5262 5284 5262
q6 218 128 133 128
q7 2013 1671 1679 1671
q8 3129 3304 3315 3304
q9 8440 8390 8452 8390
q10 3345 3181 3150 3150
q11 581 487 499 487
q12 766 611 619 611
q13 16414 2960 2971 2960
q14 307 279 273 273
q15 520 471 489 471
q16 600 542 543 542
q17 1758 1506 1449 1449
q18 7780 7463 7483 7463
q19 1640 1486 1540 1486
q20 2080 1896 1850 1850
q21 7512 5179 5039 5039
q22 1098 1029 993 993
Total cold run time: 73083 ms
Total hot run time: 54280 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2285221089
run compile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2283476073
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2291039722
<details>
<summary>TPC-H: <b>Total hot run time: 38603 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30e688d087e1d5277dbe34e2cdc7bfa4aa723733, data reload: false
------ Round 1 ----------------------------------
q1 18361 4453 4296 4296
q2 2067 233 248 233
q3 11145 1087 1145 1087
q4 10506 789 730 730
q5 7790 2860 2882 2860
q6 263 154 156 154
q7 1005 649 638 638
q8 9395 2136 2110 2110
q9 6925 6559 6617 6559
q10 7088 2260 2269 2260
q11 494 273 275 273
q12 430 265 254 254
q13 17785 3013 3011 3011
q14 309 259 262 259
q15 579 529 557 529
q16 539 420 427 420
q17 992 730 728 728
q18 7571 6884 6723 6723
q19 6280 1112 1151 1112
q20 725 357 375 357
q21 3851 2974 3070 2974
q22 1138 1036 1055 1036
Total cold run time: 115238 ms
Total hot run time: 38603 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4520 4299 4306 4299
q2 411 315 315 315
q3 2898 2691 2663 2663
q4 1974 1781 1715 1715
q5 5704 5667 5737 5667
q6 240 147 150 147
q7 2179 1823 1785 1785
q8 3309 3532 3468 3468
q9 8739 8750 8734 8734
q10 3625 3369 3330 3330
q11 612 510 537 510
q12 854 666 646 646
q13 16299 3160 3138 3138
q14 315 299 296 296
q15 560 517 515 515
q16 504 457 452 452
q17 1882 1551 1519 1519
q18 8178 7883 7825 7825
q19 10017 1635 1620 1620
q20 3449 1893 1889 1889
q21 14010 5344 5372 5344
q22 1185 1081 1062 1062
Total cold run time: 91464 ms
Total hot run time: 56939 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2244499674
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1687601725
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -387,11 +412,27 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
msg = "malformed version value";
break;
}
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer_->submit(
+ instance_id, version_pb.txn_id());
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ code = ret.first;
+ msg = ret.second;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "wait txn lazy commit failed, txn_id="
+ << version_pb.txn_id();
+ break;
+ }
+ goto TRY_AGAIN;
+ }
response->add_versions(version_pb.version());
response->add_version_update_time_ms(version_pb.update_time_ms());
}
}
+ if (code != MetaServiceCode::OK) break;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (code != MetaServiceCode::OK) { break;
}
```
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -387,11 +412,27 @@
msg = "malformed version value";
break;
}
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer_->submit(
+ instance_id, version_pb.txn_id());
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ code = ret.first;
+ msg = ret.second;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "wait txn lazy commit failed, txn_id="
+ << version_pb.txn_id();
+ break;
+ }
+ goto TRY_AGAIN;
+ }
response->add_versions(version_pb.version());
response->add_version_update_time_ms(version_pb.update_time_ms());
}
}
+ if (code != MetaServiceCode::OK) break;
}
+ if (code != MetaServiceCode::OK) break;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (code != MetaServiceCode::OK) { break;
}
```
##########
cloud/src/meta-service/txn_lazy_committer.h:
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <atomic>
+
+#include "common/config.h"
Review Comment:
warning: 'common/config.h' file not found [clang-diagnostic-error]
```cpp
#include "common/config.h"
^
```
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "meta-service/txn_lazy_committer.h"
Review Comment:
warning: 'meta-service/txn_lazy_committer.h' file not found [clang-diagnostic-error]
```cpp
#include "meta-service/txn_lazy_committer.h"
^
```
##########
cloud/src/recycler/recycler_service.h:
##########
@@ -20,6 +20,7 @@
#include <gen_cpp/cloud.pb.h>
Review Comment:
warning: 'gen_cpp/cloud.pb.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/cloud.pb.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2259875139
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2249765436
<details>
<summary>TPC-DS: <b>Total hot run time: 173437 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf6aad6e7d1b3229f6fdc1b41de15dc9b7d01bfc, data reload: false
query1 906 365 362 362
query2 6420 2050 1947 1947
query3 6633 206 227 206
query4 26326 17597 17202 17202
query5 3582 485 478 478
query6 286 176 179 176
query7 4585 297 286 286
query8 250 193 191 191
query9 8529 2417 2385 2385
query10 438 291 276 276
query11 12568 10058 9939 9939
query12 120 91 84 84
query13 1654 372 370 370
query14 10167 7406 7671 7406
query15 208 161 164 161
query16 7673 462 443 443
query17 1378 532 543 532
query18 1933 281 273 273
query19 199 146 162 146
query20 94 87 79 79
query21 209 128 123 123
query22 4407 4008 4078 4008
query23 34386 33495 33531 33495
query24 11288 3014 2979 2979
query25 593 384 448 384
query26 774 157 158 157
query27 2210 285 274 274
query28 6128 2087 2073 2073
query29 917 632 638 632
query30 260 160 150 150
query31 979 783 777 777
query32 100 56 54 54
query33 767 331 347 331
query34 933 488 487 487
query35 867 749 754 749
query36 1135 1006 957 957
query37 140 81 82 81
query38 2968 2903 2809 2809
query39 901 874 829 829
query40 202 123 116 116
query41 48 45 46 45
query42 118 98 93 93
query43 527 470 483 470
query44 1191 727 758 727
query45 210 172 172 172
query46 1083 732 728 728
query47 1857 1785 1768 1768
query48 365 292 287 287
query49 819 403 412 403
query50 783 387 390 387
query51 6813 6626 6693 6626
query52 102 89 91 89
query53 356 290 292 290
query54 901 446 447 446
query55 76 77 74 74
query56 278 292 263 263
query57 1137 1060 1042 1042
query58 248 245 254 245
query59 2920 2654 2730 2654
query60 302 279 272 272
query61 99 105 96 96
query62 824 655 634 634
query63 319 283 286 283
query64 9137 2206 1623 1623
query65 3170 3121 3096 3096
query66 741 333 337 333
query67 15501 15031 15108 15031
query68 5317 561 540 540
query69 654 433 366 366
query70 1112 1158 1087 1087
query71 455 283 274 274
query72 7331 5968 5302 5302
query73 769 322 330 322
query74 6182 5706 5665 5665
query75 3682 2697 2689 2689
query76 3156 963 875 875
query77 640 319 292 292
query78 10358 9141 9095 9095
query79 1633 539 529 529
query80 2441 563 486 486
query81 595 227 227 227
query82 427 137 144 137
query83 305 169 165 165
query84 265 86 95 86
query85 1061 309 300 300
query86 438 338 313 313
query87 3269 3099 3147 3099
query88 3368 2386 2364 2364
query89 471 391 412 391
query90 1637 197 200 197
query91 137 100 102 100
query92 65 51 50 50
query93 1951 535 525 525
query94 796 305 298 298
query95 415 332 323 323
query96 596 274 279 274
query97 3187 3042 3030 3030
query98 228 196 195 195
query99 1568 1281 1257 1257
Total cold run time: 278547 ms
Total hot run time: 173437 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1698220600
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:51:** 117 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2260238093
<details>
<summary>ClickBench: <b>Total hot run time: 30.67 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae48134efcfe7887d2b0b6f2d645e9c4fac57e59, data reload: false
query1 0.04 0.04 0.04
query2 0.07 0.04 0.04
query3 0.22 0.05 0.05
query4 1.69 0.06 0.06
query5 0.49 0.48 0.49
query6 1.14 0.72 0.71
query7 0.02 0.01 0.01
query8 0.06 0.04 0.04
query9 0.60 0.52 0.54
query10 0.58 0.59 0.60
query11 0.17 0.12 0.12
query12 0.15 0.13 0.12
query13 0.62 0.60 0.59
query14 0.78 0.79 0.80
query15 0.90 0.85 0.84
query16 0.36 0.35 0.36
query17 1.00 0.99 1.00
query18 0.22 0.22 0.21
query19 1.79 1.72 1.80
query20 0.01 0.03 0.01
query21 15.40 0.78 0.66
query22 4.32 7.24 2.00
query23 18.03 1.34 1.30
query24 2.26 0.22 0.22
query25 0.19 0.09 0.08
query26 0.31 0.22 0.22
query27 0.46 0.24 0.23
query28 13.16 0.97 0.97
query29 12.59 3.29 3.26
query30 0.26 0.06 0.06
query31 2.86 0.41 0.40
query32 3.28 0.48 0.48
query33 2.96 2.94 2.98
query34 15.50 4.26 4.26
query35 4.27 4.26 4.27
query36 0.68 0.48 0.48
query37 0.19 0.17 0.16
query38 0.16 0.15 0.15
query39 0.05 0.03 0.03
query40 0.15 0.12 0.12
query41 0.09 0.04 0.05
query42 0.06 0.05 0.04
query43 0.04 0.03 0.04
Total cold run time: 108.18 s
Total hot run time: 30.67 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2261893023
<details>
<summary>TPC-H: <b>Total hot run time: 41784 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c0a6c5ac10a4aa5587bfabcd3b74e29aa97bef62, data reload: false
------ Round 1 ----------------------------------
q1 17655 4190 4122 4122
q2 2036 198 208 198
q3 10547 1315 1378 1315
q4 10181 808 948 808
q5 7606 3002 2989 2989
q6 226 143 143 143
q7 1037 618 607 607
q8 9442 1922 1952 1922
q9 8504 6621 6656 6621
q10 8738 3838 3856 3838
q11 436 255 251 251
q12 415 233 234 233
q13 17776 2911 2951 2911
q14 272 243 246 243
q15 524 490 497 490
q16 522 401 380 380
q17 972 951 939 939
q18 8104 7395 7248 7248
q19 1778 1196 1228 1196
q20 566 333 334 333
q21 5335 4823 4721 4721
q22 351 282 276 276
Total cold run time: 113023 ms
Total hot run time: 41784 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4082 4003 4027 4003
q2 334 240 219 219
q3 3020 3023 3165 3023
q4 2008 2028 1932 1932
q5 5676 5463 5412 5412
q6 220 130 133 130
q7 2137 1804 1810 1804
q8 3331 3416 3365 3365
q9 8664 8739 8862 8739
q10 3971 4068 3952 3952
q11 563 465 472 465
q12 763 612 629 612
q13 16323 3106 3149 3106
q14 308 268 291 268
q15 531 501 486 486
q16 467 419 402 402
q17 1796 1732 1726 1726
q18 8344 7833 8055 7833
q19 3398 1773 1760 1760
q20 2055 1824 1827 1824
q21 5740 5553 5205 5205
q22 547 494 463 463
Total cold run time: 74278 ms
Total hot run time: 56729 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2297924866
<details>
<summary>TPC-H: <b>Total hot run time: 38216 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43feec09f2a80e3fe7f3b07ebc1b5b49bdf0e0bf, data reload: false
------ Round 1 ----------------------------------
q1 19389 5455 4311 4311
q2 2085 210 207 207
q3 11730 980 1146 980
q4 10491 803 772 772
q5 7763 2878 2790 2790
q6 262 157 154 154
q7 1037 657 653 653
q8 9396 2101 2107 2101
q9 7347 6562 6604 6562
q10 7044 2140 2196 2140
q11 454 260 268 260
q12 429 243 250 243
q13 18818 3027 3000 3000
q14 304 254 251 251
q15 553 508 540 508
q16 514 415 402 402
q17 987 653 691 653
q18 7593 7015 7000 7000
q19 6247 1145 1100 1100
q20 689 349 355 349
q21 3888 3081 2760 2760
q22 1133 1073 1020 1020
Total cold run time: 118153 ms
Total hot run time: 38216 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4555 4333 4289 4289
q2 403 309 291 291
q3 2835 2739 2727 2727
q4 2023 1723 1758 1723
q5 5661 5780 5681 5681
q6 245 153 153 153
q7 2211 1767 1771 1767
q8 3348 3451 3454 3451
q9 8852 8739 8773 8739
q10 3652 3372 3334 3334
q11 640 529 523 523
q12 830 700 649 649
q13 17211 3213 3074 3074
q14 320 294 288 288
q15 554 510 527 510
q16 510 449 461 449
q17 1845 1562 1530 1530
q18 8274 8057 7697 7697
q19 3869 1488 1663 1488
q20 2288 1907 1876 1876
q21 14662 5351 5314 5314
q22 1206 1120 1092 1092
Total cold run time: 85994 ms
Total hot run time: 56645 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2297864583
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1714974008
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
Review Comment:
warning: function 'convert_tmp_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void convert_tmp_rowsets(
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:36:** 152 lines including whitespace and comments (threshold 80)
```cpp
void convert_tmp_rowsets(
^
```
</details>
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:208:** 122 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1716269904
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -969,320 +1356,345 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
- msg = ss.str();
- return;
- }
- version = version_pb.version();
- } else {
- version = 1;
- }
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
-
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ if (request->has_is_2pc() && request->is_2pc()) {
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
-
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
-
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
return;
}
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- // Save table versions
- for (auto& i : table_id_tablet_ids) {
- std::string ver_key = table_version_key({instance_id, db_id, i.first});
- txn->atomic_add(ver_key, 1);
- LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key) << " txn_id=" << txn_id;
- }
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
+ }
- LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
+ // save versions for partition
+ response->clear_table_ids();
+ response->clear_partition_ids();
+ response->clear_versions();
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- // Update txn_info
- txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id;
- auto now_time = system_clock::now();
- uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
- if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
- LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
- << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
- return;
- }
- txn_info.set_commit_time(commit_time);
- txn_info.set_finish_time(commit_time);
- if (request->has_commit_attachment()) {
- txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
- }
- LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
- info_val.clear();
- if (!txn_info.SerializeToString(&info_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(info_key, info_val);
- LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
- // Update stats of affected tablet
- for (auto& [tablet_id, stats] : tablet_stats) {
- DCHECK(tablet_ids.count(tablet_id));
- auto& tablet_idx = tablet_ids[tablet_id];
- StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
- tablet_idx.partition_id(), tablet_id};
- update_tablet_stats(info, stats, txn, code, msg);
- if (code != MetaServiceCode::OK) return;
- }
- // Remove tmp rowset meta
- for (auto& [k, _] : tmp_rowsets_meta) {
- txn->remove(k);
- LOG(INFO) << "xxx remove tmp_rowset_key=" << hex(k) << " txn_id=" << txn_id;
- }
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << "txn_id=" << txn_id << " table_id=" << table_id
+ << " partition_id=" << partition_id << " version=" << i.second;
- const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
- LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
- txn->remove(running_key);
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
- std::string recycle_val;
- std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
- RecycleTxnPB recycle_pb;
- recycle_pb.set_creation_time(commit_time);
- recycle_pb.set_label(txn_info.label());
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- if (!recycle_pb.SerializeToString(&recycle_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(recycle_key, recycle_val);
+ for (size_t i = 0; i < lock_keys.size(); i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- if (txn_info.load_job_source_type() ==
- LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
- put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
- }
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
- LOG(INFO) << "xxx commit_txn put recycle_key key=" << hex(recycle_key) << " txn_id=" << txn_id;
- LOG(INFO) << "commit_txn put_size=" << txn->put_bytes() << " del_size=" << txn->delete_bytes()
- << " num_put_keys=" << txn->num_put_keys() << " num_del_keys=" << txn->num_del_keys()
- << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
- // Finally we are done...
- err = txn->commit();
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::COMMIT>(err);
- ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- return;
- }
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
+ << " txn_id=" << txn_id;
+ }
- // calculate table stats from tablets stats
- std::map<int64_t /*table_id*/, TableStats> table_stats;
- std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
- request->base_tablet_ids().end());
- calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
- for (const auto& pair : table_stats) {
- TableStatsPB* stats_pb = response->add_table_stats();
- auto table_id = pair.first;
- auto stats = pair.second;
- get_pb_from_tablestats(stats, stats_pb);
- stats_pb->set_table_id(table_id);
- VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
- << " table_id=" << table_id
- << " updated_row_count=" << stats_pb->updated_row_count();
- }
+ LOG(INFO) << "commit_txn_eventually put_size=" << txn->put_bytes()
+ << " del_size=" << txn->delete_bytes() << " num_put_keys=" << txn->num_put_keys()
+ << " num_del_keys=" << txn->num_del_keys()
+ << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
- response->mutable_txn_info()->CopyFrom(txn_info);
-} // end commit_txn
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer->submit(instance_id, txn_id);
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
Review Comment:
Some fields of the `CommitTxnResponse` are missing.
- `version_update_time_ms`
- `table_stats`
- `txn_info`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1706546958
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -788,6 +788,164 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
txn->put(key, val);
}
}
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
Review Comment:
> Add a debug log.
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2284135402
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2300412342
<details>
<summary>TPC-H: <b>Total hot run time: 38493 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1d63b327147196a6029b3421305f1eed1a00dc18, data reload: false
------ Round 1 ----------------------------------
q1 18478 5211 4311 4311
q2 2066 218 225 218
q3 11699 974 1110 974
q4 10518 713 734 713
q5 7805 2868 2864 2864
q6 263 159 160 159
q7 1066 661 665 661
q8 9624 2135 2114 2114
q9 8478 6578 6579 6578
q10 7054 2265 2227 2227
q11 494 273 276 273
q12 435 279 260 260
q13 17783 2992 3034 2992
q14 291 268 273 268
q15 574 534 534 534
q16 537 420 417 417
q17 1004 808 746 746
q18 7650 6871 6793 6793
q19 6505 1055 1130 1055
q20 727 366 365 365
q21 4332 3019 2914 2914
q22 1157 1057 1087 1057
Total cold run time: 118540 ms
Total hot run time: 38493 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4617 4320 4307 4307
q2 414 324 309 309
q3 2866 2711 2741 2711
q4 2030 1817 1734 1734
q5 5611 5795 5605 5605
q6 241 159 149 149
q7 2168 1768 1777 1768
q8 3409 3489 3456 3456
q9 8853 8769 8863 8769
q10 3583 3326 3250 3250
q11 617 520 526 520
q12 876 688 667 667
q13 17378 3186 3111 3111
q14 332 327 305 305
q15 581 521 523 521
q16 534 464 494 464
q17 1865 1561 1544 1544
q18 8473 8044 7688 7688
q19 10079 1681 1604 1604
q20 2192 1915 1882 1882
q21 13459 5357 5385 5357
q22 1137 1114 1060 1060
Total cold run time: 91315 ms
Total hot run time: 56781 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2297939772
<details>
<summary>ClickBench: <b>Total hot run time: 30.75 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43feec09f2a80e3fe7f3b07ebc1b5b49bdf0e0bf, data reload: false
query1 0.05 0.05 0.04
query2 0.07 0.05 0.05
query3 0.23 0.06 0.05
query4 1.67 0.07 0.07
query5 0.48 0.48 0.48
query6 1.13 0.73 0.73
query7 0.03 0.02 0.02
query8 0.05 0.05 0.06
query9 0.55 0.50 0.49
query10 0.56 0.55 0.57
query11 0.16 0.12 0.12
query12 0.16 0.13 0.12
query13 0.62 0.62 0.60
query14 0.78 0.79 0.79
query15 0.85 0.84 0.84
query16 0.35 0.39 0.36
query17 1.06 0.99 0.98
query18 0.22 0.21 0.21
query19 1.86 1.75 1.75
query20 0.02 0.01 0.01
query21 15.40 0.67 0.68
query22 4.19 8.02 1.56
query23 18.33 1.38 1.27
query24 2.06 0.22 0.24
query25 0.16 0.08 0.09
query26 0.27 0.19 0.18
query27 0.08 0.08 0.09
query28 13.22 1.03 1.00
query29 12.64 3.31 3.32
query30 0.43 0.24 0.23
query31 2.80 0.40 0.40
query32 3.25 0.48 0.49
query33 3.00 2.99 2.93
query34 16.82 4.40 4.40
query35 4.47 4.45 4.50
query36 0.67 0.48 0.48
query37 0.20 0.17 0.17
query38 0.18 0.16 0.17
query39 0.06 0.05 0.05
query40 0.17 0.14 0.15
query41 0.11 0.07 0.06
query42 0.07 0.06 0.07
query43 0.07 0.06 0.06
Total cold run time: 109.55 s
Total hot run time: 30.75 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312021091
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312692856
<details>
<summary>TPC-DS: <b>Total hot run time: 191755 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5f3b5ea8bfb6214887e8d1bb42d8f87233ccc8a0, data reload: false
query1 1263 887 874 874
query2 6324 1975 1987 1975
query3 10599 3842 3974 3842
query4 59298 25088 23183 23183
query5 5434 500 509 500
query6 406 165 169 165
query7 5855 297 306 297
query8 298 212 201 201
query9 8981 2496 2446 2446
query10 486 294 269 269
query11 18335 15007 15251 15007
query12 156 101 103 101
query13 1573 418 383 383
query14 11066 6494 7551 6494
query15 256 178 183 178
query16 7517 465 483 465
query17 1132 560 555 555
query18 1897 294 303 294
query19 301 150 148 148
query20 122 109 121 109
query21 207 106 102 102
query22 4667 4361 4299 4299
query23 34328 33420 33373 33373
query24 5948 2881 2845 2845
query25 552 416 416 416
query26 688 165 167 165
query27 1785 285 282 282
query28 3702 2077 2063 2063
query29 712 425 434 425
query30 240 154 151 151
query31 917 754 763 754
query32 81 56 59 56
query33 464 292 293 292
query34 867 491 506 491
query35 846 712 720 712
query36 1083 899 969 899
query37 138 90 86 86
query38 4039 3882 3897 3882
query39 1444 1388 1413 1388
query40 190 113 116 113
query41 49 44 44 44
query42 113 96 93 93
query43 493 468 464 464
query44 1111 760 745 745
query45 200 167 166 166
query46 1108 739 747 739
query47 1861 1803 1792 1792
query48 385 303 304 303
query49 758 421 430 421
query50 828 416 433 416
query51 7102 7167 7076 7076
query52 100 86 87 86
query53 253 183 182 182
query54 588 452 461 452
query55 74 75 76 75
query56 284 262 264 262
query57 1179 1057 1096 1057
query58 227 258 230 230
query59 3140 2744 2738 2738
query60 316 278 296 278
query61 100 97 104 97
query62 754 662 672 662
query63 217 182 181 181
query64 2843 680 652 652
query65 3216 3169 3177 3169
query66 694 333 332 332
query67 15797 15109 15203 15109
query68 2989 576 575 575
query69 389 328 284 284
query70 1138 1107 1016 1016
query71 345 281 272 272
query72 6131 4019 4000 4000
query73 763 336 339 336
query74 9139 8796 8898 8796
query75 3357 2714 2727 2714
query76 1500 998 953 953
query77 527 325 317 317
query78 11320 9262 9015 9015
query79 1048 546 536 536
query80 704 518 521 518
query81 458 229 230 229
query82 226 136 133 133
query83 170 149 148 148
query84 264 78 82 78
query85 681 345 277 277
query86 304 297 307 297
query87 4380 4256 4231 4231
query88 3239 2403 2392 2392
query89 384 288 282 282
query90 1905 194 203 194
query91 122 97 102 97
query92 60 52 55 52
query93 1048 537 546 537
query94 593 301 309 301
query95 352 274 268 268
query96 598 276 272 272
query97 3158 3049 3071 3049
query98 219 206 200 200
query99 1514 1306 1285 1285
Total cold run time: 307012 ms
Total hot run time: 191755 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2302291594
<details>
<summary>TPC-H: <b>Total hot run time: 38859 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ecb308aa0ea6ae75918b441fcce5ed28f71c479, data reload: false
------ Round 1 ----------------------------------
q1 18559 4793 4337 4337
q2 2056 220 236 220
q3 11688 1016 1208 1016
q4 10488 754 742 742
q5 7787 2933 2863 2863
q6 273 159 162 159
q7 1039 700 660 660
q8 9394 2166 2162 2162
q9 7098 6564 6607 6564
q10 7079 2276 2265 2265
q11 501 283 286 283
q12 448 273 262 262
q13 18851 3030 3053 3030
q14 316 265 263 263
q15 578 534 524 524
q16 554 445 421 421
q17 1022 649 793 649
q18 7510 6855 6971 6855
q19 1465 1203 1127 1127
q20 755 367 365 365
q21 4127 3016 3155 3016
q22 1176 1091 1076 1076
Total cold run time: 112764 ms
Total hot run time: 38859 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4390 4359 4330 4330
q2 412 333 320 320
q3 2947 2745 2696 2696
q4 1970 1704 1806 1704
q5 5688 5708 5748 5708
q6 234 149 155 149
q7 2292 1894 1859 1859
q8 3280 3468 3536 3468
q9 8841 8869 8799 8799
q10 3612 3434 3419 3419
q11 633 521 555 521
q12 895 662 707 662
q13 15934 3278 3240 3240
q14 334 317 307 307
q15 561 525 513 513
q16 506 439 440 439
q17 1839 1560 1559 1559
q18 8364 7810 7879 7810
q19 1821 1637 1490 1490
q20 2135 1933 1958 1933
q21 5739 5666 5514 5514
q22 1186 1156 1114 1114
Total cold run time: 73613 ms
Total hot run time: 57554 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2274972042
<details>
<summary>TPC-H: <b>Total hot run time: 40202 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit faf1651134a2bd8b1faa3c32beaf000445742a69, data reload: false
------ Round 1 ----------------------------------
q1 19152 4713 4380 4380
q2 2024 184 181 181
q3 10534 1151 1117 1117
q4 10171 756 709 709
q5 7517 2592 2568 2568
q6 232 140 141 140
q7 984 628 593 593
q8 9231 1956 1956 1956
q9 8830 6601 6593 6593
q10 7071 2174 2203 2174
q11 467 246 255 246
q12 413 233 228 228
q13 18924 3000 3010 3000
q14 288 248 240 240
q15 531 484 502 484
q16 640 520 527 520
q17 1009 715 707 707
q18 8142 7629 7438 7438
q19 3670 1052 1094 1052
q20 757 433 435 433
q21 5629 4451 4623 4451
q22 1102 1005 992 992
Total cold run time: 117318 ms
Total hot run time: 40202 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4436 4285 4252 4252
q2 377 278 269 269
q3 2882 2585 2673 2585
q4 1915 1590 1593 1590
q5 5289 5269 5307 5269
q6 233 132 136 132
q7 2113 1676 1692 1676
q8 3206 3396 3361 3361
q9 8493 8488 8442 8442
q10 3435 3189 3104 3104
q11 589 498 498 498
q12 777 612 591 591
q13 16656 2987 2975 2975
q14 304 287 264 264
q15 523 476 476 476
q16 609 555 544 544
q17 1796 1515 1484 1484
q18 7721 7954 7529 7529
q19 1697 1671 1587 1587
q20 2085 1925 1871 1871
q21 8551 5133 5080 5080
q22 1089 1039 1079 1039
Total cold run time: 74776 ms
Total hot run time: 54618 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2275027140
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1730850089
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
Review Comment:
> and it should be a snapshot GET, concurrent updating does not matter, otherwise the update procedure may no succeed if there are other reads on tablet_idx.
it maybe conflict with recycler, cannot snapshot read
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2317507906
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2272741527
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1706688019
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -387,10 +410,28 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
msg = "malformed version value";
break;
}
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
+ std::tie(code, msg) = task->wait();
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "wait txn lazy commit failed, txn_id="
+ << version_pb.txn_id();
+ break;
+ }
+ goto TRY_AGAIN;
Review Comment:
We don't need to retry again, just like the `get_version`, returning the version directly is enough.
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -237,46 +238,63 @@ void MetaServiceImpl::get_version(::google::protobuf::RpcController* controller,
partition_version_key({instance_id, db_id, table_id, partition_id}, &ver_key);
}
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv_->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- msg = "failed to create txn";
- code = cast_as<ErrCategory::CREATE>(err);
- return;
- }
+ do {
+ code = MetaServiceCode::OK;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ msg = "failed to create txn";
+ code = cast_as<ErrCategory::CREATE>(err);
+ return;
+ }
- std::string ver_val;
- // 0 for success get a key, 1 for key not found, negative for error
- err = txn->get(ver_key, &ver_val);
- VLOG_DEBUG << "xxx get version_key=" << hex(ver_key);
- if (err == TxnErrorCode::TXN_OK) {
- if (is_table_version) {
- int64_t version = 0;
- if (!txn->decode_atomic_int(ver_val, &version)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "malformed table version value";
- return;
- }
- response->set_version(version);
- } else {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(ver_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "malformed version value";
- return;
+ std::string ver_val;
+ // 0 for success get a key, 1 for key not found, negative for error
+ err = txn->get(ver_key, &ver_val);
+ VLOG_DEBUG << "xxx get version_key=" << hex(ver_key);
+ if (err == TxnErrorCode::TXN_OK) {
+ if (is_table_version) {
+ int64_t version = 0;
+ if (!txn->decode_atomic_int(ver_val, &version)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "malformed table version value";
+ return;
+ }
+ response->set_version(version);
+ } else {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "malformed version value";
+ return;
+ }
+
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
+
+ std::tie(code, msg) = task->wait();
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING)
+ << "wait txn lazy commit failed, txn_id=" << version_pb.txn_id();
+ return;
+ }
+ }
+
+ response->set_version(version_pb.version());
+ response->add_version_update_time_ms(version_pb.update_time_ms());
}
- response->set_version(version_pb.version());
- response->add_version_update_time_ms(version_pb.update_time_ms());
+ { TEST_SYNC_POINT_CALLBACK("get_version_code", &code); }
+ return;
+ } else if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ msg = "not found";
+ code = MetaServiceCode::VERSION_NOT_FOUND;
+ return;
}
- { TEST_SYNC_POINT_CALLBACK("get_version_code", &code); }
- return;
- } else if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- msg = "not found";
- code = MetaServiceCode::VERSION_NOT_FOUND;
- return;
- }
- msg = fmt::format("failed to get txn, err={}", err);
- code = cast_as<ErrCategory::READ>(err);
+ msg = fmt::format("failed to get txn, err={}", err);
+ code = cast_as<ErrCategory::READ>(err);
+ } while (false);
Review Comment:
remove the `while`, since the `continue` are removed.
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ break;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t, std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end = (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id_, txn_id_, txn_kv_, code_, msg_, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code_ != MetaServiceCode::OK) break;
+ }
+ if (code_ != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id_, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code_ = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id_
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id_ << " key=" << hex(ver_key);
+ msg_ = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id_) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id_;
+ msg_ = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
Review Comment:
Handle `TxnErrorCode::TXN_CONFLICT` too.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -917,320 +1461,418 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
- msg = ss.str();
- return;
- }
- version = version_pb.version();
- } else {
- version = 1;
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
}
- new_versions[version_keys[i]] = version + 1;
+ txn->put(recycle_key, recycle_val);
}
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ err = txn->commit();
Review Comment:
No need to commit if this TXN is in TXN_STATUS_VISIBLE
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ break;
Review Comment:
Even if the tmp_rowset_metas is empty, the TXN might still not be visible. Consider call `make_committed_txn_visible` here.
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -387,10 +410,28 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
msg = "malformed version value";
break;
}
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
+ std::tie(code, msg) = task->wait();
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "wait txn lazy commit failed, txn_id="
+ << version_pb.txn_id();
+ break;
+ }
+ goto TRY_AGAIN;
+ }
response->add_versions(version_pb.version());
response->add_version_update_time_ms(version_pb.update_time_ms());
}
}
+ if (code != MetaServiceCode::OK) {
+ break;
+ }
+ }
+ if (code != MetaServiceCode::OK) {
+ break;
Review Comment:
Delete the useless code
##########
cloud/src/recycler/recycler_service.cpp:
##########
@@ -42,8 +42,12 @@ extern int reset_s3_rate_limiter(S3RateLimitType type, size_t max_speed, size_t
extern std::tuple<int, std::string_view> convert_ms_code_to_http_code(MetaServiceCode ret);
RecyclerServiceImpl::RecyclerServiceImpl(std::shared_ptr<TxnKv> txn_kv, Recycler* recycler,
- Checker* checker)
- : txn_kv_(std::move(txn_kv)), recycler_(recycler), checker_(checker) {}
+ Checker* checker,
+ std::shared_ptr<TxnLazyCommitter> txn_lazy_committer)
+ : txn_kv_(std::move(txn_kv)),
+ recycler_(recycler),
+ checker_(checker),
+ txn_lazy_committer_(txn_lazy_committer) {}
Review Comment:
```suggestion
txn_lazy_committer_(std::move(txn_lazy_committer)) {}
```
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -917,320 +1461,418 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
- msg = ss.str();
- return;
- }
- version = version_pb.version();
- } else {
- version = 1;
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
}
- new_versions[version_keys[i]] = version + 1;
+ txn->put(recycle_key, recycle_val);
}
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
Review Comment:
It seems that there is no need to read the tablet index again during retry.
Merge the PR first and then optimize this issue later.
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ break;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t, std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end = (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id_, txn_id_, txn_kv_, code_, msg_, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code_ != MetaServiceCode::OK) break;
+ }
+ if (code_ != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id_, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code_ = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id_
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id_ << " key=" << hex(ver_key);
+ msg_ = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id_) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id_;
+ msg_ = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+ }
+
+ err = txn->commit();
Review Comment:
No need to commit if `version_pb.has_txn_id() && version_pb.txn_id() == txn_id_` is false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2275189325
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2291136442
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1732038003
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -862,94 +863,485 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
std::stringstream ss;
int64_t txn_id = request->txn_id();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // Get txn info with db_id and txn_id
- std::string info_val; // Will be reused when saving updated txn
- const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
- err = txn->get(info_key, &info_val);
- if (err != TxnErrorCode::TXN_OK) {
- code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
- : cast_as<ErrCategory::READ>(err);
- if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
- } else {
- ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
- << " err=" << err;
+ // Get txn info with db_id and txn_id
+ std::string info_val; // Will be reused when saving updated txn
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
+ } else {
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
+ }
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(info_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // TODO: do more check like txn state, 2PC etc.
- DCHECK(txn_info.txn_id() == txn_id);
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
- code = MetaServiceCode::TXN_ALREADY_ABORTED;
- ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ // TODO: do more check like txn state, 2PC etc.
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
- if (request->has_is_2pc() && request->is_2pc()) {
- code = MetaServiceCode::TXN_ALREADY_VISIBLE;
- ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(INFO) << msg;
response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- code = MetaServiceCode::OK;
- ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(INFO) << msg;
- response->mutable_txn_info()->CopyFrom(txn_info);
- return;
- }
- if (request->has_is_2pc() && request->is_2pc() &&
- txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
- code = MetaServiceCode::TXN_INVALID_STATUS;
- ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Prepare rowset meta and new_versions
+ // Prepare rowset meta and new_versions
+ // Read tablet indexes in batch.
+ std::vector<std::string> tablet_idx_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
+ }
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ size_t total_rowsets = tmp_rowsets_meta.size();
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ for (size_t i = 0; i < total_rowsets; i++) {
+ uint64_t tablet_id = tmp_rowsets_meta[i].second.tablet_id();
+ if (!tablet_idx_values[i].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " tablet_id=" << tablet_id << " key=" << hex(tablet_idx_keys[i]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "malformed tablet index value tablet_id=" << tablet_id
Review Comment:
> do not change what you don't modify
for pipeline code format check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1713021124
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -265,6 +267,20 @@ void MetaServiceImpl::get_version(::google::protobuf::RpcController* controller,
msg = "malformed version value";
return;
}
+
+ if (version_pb.has_txn_id()) {
+ txn.reset();
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer_->submit(instance_id, version_pb.txn_id());
Review Comment:
how do we deal with the "duplicated" submmits to to the commiter
add comment here.
##########
gensrc/proto/cloud.proto:
##########
@@ -666,6 +667,7 @@ message CommitTxnRequest {
// for transaction load
optional bool is_txn_load = 9;
repeated SubTxnInfo sub_txn_infos = 10;
+ optional bool lazy_commit = 11;
Review Comment:
naming `enable_lazy_commit`
##########
be/src/cloud/config.h:
##########
@@ -94,4 +94,7 @@ DECLARE_mBool(save_load_error_log_to_s3);
// the theads which sync the datas which loaded in other clusters
DECLARE_mInt32(sync_load_for_tablets_thread);
+// enable large txn lazy commit in meta-service `commit_txn`
+DECLARE_mBool(cloud_txn_lazy_commit);
Review Comment:
naming `enable_cloud_txn_lazy_commit`
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -1420,104 +1454,152 @@ void MetaServiceImpl::get_rowset(::google::protobuf::RpcController* controller,
int64_t req_cc_cnt = request->cumulative_compaction_cnt();
int64_t req_cp = request->cumulative_point();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv_->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- msg = "failed to create txn";
- return;
- }
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ msg = "failed to create txn";
+ LOG(WARNING) << msg;
+ return;
+ }
- TabletIndexPB idx(request->idx());
- // Get tablet id index from kv
- if (!idx.has_table_id() || !idx.has_index_id() || !idx.has_partition_id()) {
+ TabletIndexPB idx;
+ // Get tablet id index from kv
get_tablet_idx(code, msg, txn.get(), instance_id, tablet_id, idx);
- if (code != MetaServiceCode::OK) return;
- }
- // TODO(plat1ko): Judge if tablet has been dropped (in dropped index/partition)
-
- TabletStatsPB tablet_stat;
- internal_get_tablet_stats(code, msg, txn.get(), instance_id, idx, tablet_stat, true);
- if (code != MetaServiceCode::OK) return;
- VLOG_DEBUG << "tablet_id=" << tablet_id << " stats=" << proto_to_json(tablet_stat);
+ if (code != MetaServiceCode::OK) {
+ return;
+ }
+ DCHECK(request->has_idx());
- int64_t bc_cnt = tablet_stat.base_compaction_cnt();
- int64_t cc_cnt = tablet_stat.cumulative_compaction_cnt();
- int64_t cp = tablet_stat.cumulative_point();
+ if (idx.has_db_id()) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, idx.db_id(), idx.table_id(), idx.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err && TxnErrorCode::TXN_KEY_NOT_FOUND != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, tablet_id=" << tablet_id
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- response->mutable_stats()->CopyFrom(tablet_stat);
+ if (TxnErrorCode::TXN_OK == err) {
Review Comment:
is there any chance we get other errors and do we have to deal with them?
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
Review Comment:
use constexpr instead of MACRO, and naming should be
`MAX_NUM_TABLET_INDEX`
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -862,94 +863,485 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
std::stringstream ss;
int64_t txn_id = request->txn_id();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // Get txn info with db_id and txn_id
- std::string info_val; // Will be reused when saving updated txn
- const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
- err = txn->get(info_key, &info_val);
- if (err != TxnErrorCode::TXN_OK) {
- code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
- : cast_as<ErrCategory::READ>(err);
- if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
- } else {
- ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
- << " err=" << err;
+ // Get txn info with db_id and txn_id
+ std::string info_val; // Will be reused when saving updated txn
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
+ } else {
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
+ }
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(info_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // TODO: do more check like txn state, 2PC etc.
- DCHECK(txn_info.txn_id() == txn_id);
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
- code = MetaServiceCode::TXN_ALREADY_ABORTED;
- ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ // TODO: do more check like txn state, 2PC etc.
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
- if (request->has_is_2pc() && request->is_2pc()) {
- code = MetaServiceCode::TXN_ALREADY_VISIBLE;
- ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(INFO) << msg;
response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- code = MetaServiceCode::OK;
- ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(INFO) << msg;
- response->mutable_txn_info()->CopyFrom(txn_info);
- return;
- }
- if (request->has_is_2pc() && request->is_2pc() &&
- txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
- code = MetaServiceCode::TXN_INVALID_STATUS;
- ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Prepare rowset meta and new_versions
+ // Prepare rowset meta and new_versions
+ // Read tablet indexes in batch.
+ std::vector<std::string> tablet_idx_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
+ }
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ size_t total_rowsets = tmp_rowsets_meta.size();
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ for (size_t i = 0; i < total_rowsets; i++) {
+ uint64_t tablet_id = tmp_rowsets_meta[i].second.tablet_id();
+ if (!tablet_idx_values[i].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " tablet_id=" << tablet_id << " key=" << hex(tablet_idx_keys[i]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "malformed tablet index value tablet_id=" << tablet_id
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ VLOG_DEBUG << "tablet_id:" << tablet_id
+ << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ }
+
+ tablet_idx_keys.clear();
+ tablet_idx_values.clear();
+
+ // {table/partition} -> version
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ size_t total_versions = version_keys.size();
+ for (size_t i = 0; i < total_versions; i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+ version_keys.clear();
+ version_values.clear();
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
+ continue;
+ }
+
+ std::vector<std::pair<std::string, std::string>> rowsets;
+ std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
+ rowsets.reserve(tmp_rowsets_meta.size());
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions[ver_key] == 0) [[unlikely]] {
+ // it is impossible.
+ code = MetaServiceCode::UNDEFINED_ERR;
+ ss << "failed to get partition version key, the target version not exists in "
+ "new_versions."
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(ERROR) << msg;
+ return;
+ }
+
+ // Update rowset version
+ int64_t new_version = new_versions[ver_key];
+ i.set_start_version(new_version);
+ i.set_end_version(new_version);
+
+ std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
+ std::string val;
+ if (!i.SerializeToString(&val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ rowsets.emplace_back(std::move(key), std::move(val));
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tablet_id];
+ stats.data_size += i.data_disk_size();
+ stats.num_rows += i.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += i.num_segments();
+ } // for tmp_rowsets_meta
+
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ size_t total_locks = lock_keys.size();
+ for (size_t i = 0; i < total_locks; i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
+
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
Review Comment:
do not change what you don't modify
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
Review Comment:
use structured binding instead of dereference with `.first` `.second`
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
Review Comment:
also explain why
##########
cloud/src/common/config.h:
##########
@@ -201,4 +201,8 @@ CONF_Validator(s3_client_http_scheme, [](const std::string& config) -> bool {
// Max retry times for object storage request
CONF_mInt64(max_s3_client_retry, "10");
+CONF_Bool(enable_txn_lazy_commit, "true");
+CONF_Int32(txn_lazy_commit_rowsets_thresold, "1024");
Review Comment:
add comment for this config
e.g. "try to commit doris txn lazily if number of involved tablets of a single txn"
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
- // Save table versions
- for (auto& i : table_id_tablet_ids) {
- std::string ver_key = table_version_key({instance_id, db_id, i.first});
- txn->atomic_add(ver_key, 1);
- LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key) << " txn_id=" << txn_id;
- }
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << "txn_id=" << txn_id << " table_id=" << table_id
+ << " partition_id=" << partition_id << " version=" << i.second;
- LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
- // Update txn_info
- txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- auto now_time = system_clock::now();
- uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
- if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
- LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
- << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
- return;
- }
- txn_info.set_commit_time(commit_time);
- txn_info.set_finish_time(commit_time);
- if (request->has_commit_attachment()) {
- txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
- }
- LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
- info_val.clear();
- if (!txn_info.SerializeToString(&info_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(info_key, info_val);
- LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+ for (size_t i = 0; i < lock_keys.size(); i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- // Update stats of affected tablet
- for (auto& [tablet_id, stats] : tablet_stats) {
- DCHECK(tablet_ids.count(tablet_id));
- auto& tablet_idx = tablet_ids[tablet_id];
- StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
- tablet_idx.partition_id(), tablet_id};
- update_tablet_stats(info, stats, txn, code, msg);
- if (code != MetaServiceCode::OK) return;
- }
- // Remove tmp rowset meta
- for (auto& [k, _] : tmp_rowsets_meta) {
- txn->remove(k);
- LOG(INFO) << "xxx remove tmp_rowset_key=" << hex(k) << " txn_id=" << txn_id;
- }
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
- const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
- LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
- txn->remove(running_key);
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
- std::string recycle_val;
- std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
- RecycleTxnPB recycle_pb;
- recycle_pb.set_creation_time(commit_time);
- recycle_pb.set_label(txn_info.label());
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
+ << " txn_id=" << txn_id;
+ }
- if (!recycle_pb.SerializeToString(&recycle_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(recycle_key, recycle_val);
+ LOG(INFO) << "commit_txn_eventually put_size=" << txn->put_bytes()
+ << " del_size=" << txn->delete_bytes() << " num_put_keys=" << txn->num_put_keys()
+ << " num_del_keys=" << txn->num_del_keys()
+ << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
- if (txn_info.load_job_source_type() ==
- LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
- put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
- }
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
- LOG(INFO) << "xxx commit_txn put recycle_key key=" << hex(recycle_key) << " txn_id=" << txn_id;
- LOG(INFO) << "commit_txn put_size=" << txn->put_bytes() << " del_size=" << txn->delete_bytes()
- << " num_put_keys=" << txn->num_put_keys() << " num_del_keys=" << txn->num_del_keys()
- << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer->submit(instance_id, txn_id);
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ if (ret.first != MetaServiceCode::OK) {
+ LOG(WARNING) << "lazy commit txn failed txn_id=" << txn_id << " code=" << ret.first
Review Comment:
what can we do if we reach this branch?
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -3017,6 +3017,10 @@ public static int metaServiceRpcRetryTimes() {
@ConfField(mutable = true, description = {"存算分离模式下commit阶段等锁超时时间，默认5s"})
public static int try_commit_lock_timeout_seconds = 5;
+
+ @ConfField(mutable = true, description = {"存算分离模式下是否开启大事务提交，默认false"})
+ public static boolean cloud_txn_lazy_commit = false;
Review Comment:
naming `enable_cloud_txn_lazy_commit`
##########
cloud/src/common/config.h:
##########
@@ -206,4 +206,8 @@ CONF_Validator(s3_client_http_scheme, [](const std::string& config) -> bool {
// Max retry times for object storage request
CONF_mInt64(max_s3_client_retry, "10");
+CONF_Bool(enable_txn_lazy_commit, "true");
+CONF_Int32(txn_lazy_commit_rowsets_thresold, "2048");
+CONF_Int32(txn_lazy_commit_num_threads, "8");
+CONF_Int32(txn_lazy_max_rowsets_per_batch, "2048");
Review Comment:
what the relation between `txn_lazy_max_rowsets_per_batch` and `txn_lazy_max_rowsets_per_batch`
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -969,337 +1357,473 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
Review Comment:
constexpr MAX_TABLET_INDEX_NUM
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -1420,104 +1454,152 @@ void MetaServiceImpl::get_rowset(::google::protobuf::RpcController* controller,
int64_t req_cc_cnt = request->cumulative_compaction_cnt();
int64_t req_cp = request->cumulative_point();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv_->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- msg = "failed to create txn";
- return;
- }
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ msg = "failed to create txn";
+ LOG(WARNING) << msg;
+ return;
+ }
- TabletIndexPB idx(request->idx());
- // Get tablet id index from kv
- if (!idx.has_table_id() || !idx.has_index_id() || !idx.has_partition_id()) {
+ TabletIndexPB idx;
+ // Get tablet id index from kv
get_tablet_idx(code, msg, txn.get(), instance_id, tablet_id, idx);
- if (code != MetaServiceCode::OK) return;
- }
- // TODO(plat1ko): Judge if tablet has been dropped (in dropped index/partition)
-
- TabletStatsPB tablet_stat;
- internal_get_tablet_stats(code, msg, txn.get(), instance_id, idx, tablet_stat, true);
- if (code != MetaServiceCode::OK) return;
- VLOG_DEBUG << "tablet_id=" << tablet_id << " stats=" << proto_to_json(tablet_stat);
+ if (code != MetaServiceCode::OK) {
+ return;
+ }
+ DCHECK(request->has_idx());
- int64_t bc_cnt = tablet_stat.base_compaction_cnt();
- int64_t cc_cnt = tablet_stat.cumulative_compaction_cnt();
- int64_t cp = tablet_stat.cumulative_point();
+ if (idx.has_db_id()) {
Review Comment:
add comment what is this code block for. e.g. historical reason or compatibility reason or something else related to lazy commit
##########
cloud/src/meta-service/txn_lazy_committer.h:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/cloud.pb.h>
+
+#include <atomic>
+
+#include "common/simple_thread_pool.h"
+#include "meta-service/txn_kv.h"
+
+namespace doris::cloud {
+
+class TxnLazyCommitter;
+
+class TxnLazyCommitTask {
+public:
+ TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer);
+ void commit();
+
+ std::pair<MetaServiceCode, std::string> wait();
+
+private:
+ friend class TxnLazyCommitter;
+
+ std::string instance_id_;
+ int64_t txn_id_;
+ std::shared_ptr<TxnKv> txn_kv_;
+ std::mutex mutex_;
+ std::condition_variable cond_;
+ bool finished_ = false;
+ MetaServiceCode code_ = MetaServiceCode::OK;
+ std::string msg_;
+ TxnLazyCommitter* txn_lazy_committer_;
+};
+
+class TxnLazyCommitter {
+public:
+ TxnLazyCommitter(std::shared_ptr<TxnKv> txn_kv);
Review Comment:
Add comment for each function describing the behaviors and the parameters
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
Review Comment:
what if the commit returns errors like
1. value exceeds max bytes
2. transaction too old
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -862,94 +863,485 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
std::stringstream ss;
int64_t txn_id = request->txn_id();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // Get txn info with db_id and txn_id
- std::string info_val; // Will be reused when saving updated txn
- const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
- err = txn->get(info_key, &info_val);
- if (err != TxnErrorCode::TXN_OK) {
- code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
- : cast_as<ErrCategory::READ>(err);
- if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
- } else {
- ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
- << " err=" << err;
+ // Get txn info with db_id and txn_id
+ std::string info_val; // Will be reused when saving updated txn
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
+ } else {
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
+ }
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(info_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // TODO: do more check like txn state, 2PC etc.
- DCHECK(txn_info.txn_id() == txn_id);
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
- code = MetaServiceCode::TXN_ALREADY_ABORTED;
- ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ // TODO: do more check like txn state, 2PC etc.
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
- if (request->has_is_2pc() && request->is_2pc()) {
- code = MetaServiceCode::TXN_ALREADY_VISIBLE;
- ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(INFO) << msg;
response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- code = MetaServiceCode::OK;
- ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(INFO) << msg;
- response->mutable_txn_info()->CopyFrom(txn_info);
- return;
- }
- if (request->has_is_2pc() && request->is_2pc() &&
- txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
- code = MetaServiceCode::TXN_INVALID_STATUS;
- ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Prepare rowset meta and new_versions
+ // Prepare rowset meta and new_versions
+ // Read tablet indexes in batch.
+ std::vector<std::string> tablet_idx_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
+ }
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ size_t total_rowsets = tmp_rowsets_meta.size();
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ for (size_t i = 0; i < total_rowsets; i++) {
+ uint64_t tablet_id = tmp_rowsets_meta[i].second.tablet_id();
+ if (!tablet_idx_values[i].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " tablet_id=" << tablet_id << " key=" << hex(tablet_idx_keys[i]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "malformed tablet index value tablet_id=" << tablet_id
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ VLOG_DEBUG << "tablet_id:" << tablet_id
+ << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ }
+
+ tablet_idx_keys.clear();
+ tablet_idx_values.clear();
+
+ // {table/partition} -> version
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
Review Comment:
do not change what you don't modify
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -1420,104 +1454,152 @@ void MetaServiceImpl::get_rowset(::google::protobuf::RpcController* controller,
int64_t req_cc_cnt = request->cumulative_compaction_cnt();
int64_t req_cp = request->cumulative_point();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv_->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- msg = "failed to create txn";
- return;
- }
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ msg = "failed to create txn";
+ LOG(WARNING) << msg;
+ return;
+ }
- TabletIndexPB idx(request->idx());
- // Get tablet id index from kv
- if (!idx.has_table_id() || !idx.has_index_id() || !idx.has_partition_id()) {
+ TabletIndexPB idx;
+ // Get tablet id index from kv
get_tablet_idx(code, msg, txn.get(), instance_id, tablet_id, idx);
- if (code != MetaServiceCode::OK) return;
- }
- // TODO(plat1ko): Judge if tablet has been dropped (in dropped index/partition)
-
- TabletStatsPB tablet_stat;
- internal_get_tablet_stats(code, msg, txn.get(), instance_id, idx, tablet_stat, true);
- if (code != MetaServiceCode::OK) return;
- VLOG_DEBUG << "tablet_id=" << tablet_id << " stats=" << proto_to_json(tablet_stat);
+ if (code != MetaServiceCode::OK) {
+ return;
+ }
+ DCHECK(request->has_idx());
- int64_t bc_cnt = tablet_stat.base_compaction_cnt();
- int64_t cc_cnt = tablet_stat.cumulative_compaction_cnt();
- int64_t cp = tablet_stat.cumulative_point();
+ if (idx.has_db_id()) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, idx.db_id(), idx.table_id(), idx.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err && TxnErrorCode::TXN_KEY_NOT_FOUND != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, tablet_id=" << tablet_id
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- response->mutable_stats()->CopyFrom(tablet_stat);
+ if (TxnErrorCode::TXN_OK == err) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb tablet_id=" << tablet_id
Review Comment:
also log partition id
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
Review Comment:
add comment what does it do and why
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -862,94 +863,485 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
std::stringstream ss;
int64_t txn_id = request->txn_id();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // Get txn info with db_id and txn_id
- std::string info_val; // Will be reused when saving updated txn
- const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
- err = txn->get(info_key, &info_val);
- if (err != TxnErrorCode::TXN_OK) {
- code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
- : cast_as<ErrCategory::READ>(err);
- if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
- } else {
- ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
- << " err=" << err;
+ // Get txn info with db_id and txn_id
+ std::string info_val; // Will be reused when saving updated txn
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
+ } else {
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
+ }
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(info_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // TODO: do more check like txn state, 2PC etc.
- DCHECK(txn_info.txn_id() == txn_id);
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
- code = MetaServiceCode::TXN_ALREADY_ABORTED;
- ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ // TODO: do more check like txn state, 2PC etc.
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
- if (request->has_is_2pc() && request->is_2pc()) {
- code = MetaServiceCode::TXN_ALREADY_VISIBLE;
- ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(INFO) << msg;
response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- code = MetaServiceCode::OK;
- ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(INFO) << msg;
- response->mutable_txn_info()->CopyFrom(txn_info);
- return;
- }
- if (request->has_is_2pc() && request->is_2pc() &&
- txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
- code = MetaServiceCode::TXN_INVALID_STATUS;
- ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Prepare rowset meta and new_versions
+ // Prepare rowset meta and new_versions
+ // Read tablet indexes in batch.
+ std::vector<std::string> tablet_idx_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
+ }
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ size_t total_rowsets = tmp_rowsets_meta.size();
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ for (size_t i = 0; i < total_rowsets; i++) {
+ uint64_t tablet_id = tmp_rowsets_meta[i].second.tablet_id();
+ if (!tablet_idx_values[i].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " tablet_id=" << tablet_id << " key=" << hex(tablet_idx_keys[i]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "malformed tablet index value tablet_id=" << tablet_id
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ VLOG_DEBUG << "tablet_id:" << tablet_id
+ << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ }
+
+ tablet_idx_keys.clear();
+ tablet_idx_values.clear();
+
+ // {table/partition} -> version
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ size_t total_versions = version_keys.size();
+ for (size_t i = 0; i < total_versions; i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
Review Comment:
use single `last_pending_txn_id` is enough to describe the logic `need_advance_last_txn` and `last_pending_txn_id`.
because, last_pending_txn_id != 0 always means need_advance_last_txn == true? != 0 always means
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -862,94 +863,485 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
std::stringstream ss;
int64_t txn_id = request->txn_id();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // Get txn info with db_id and txn_id
- std::string info_val; // Will be reused when saving updated txn
- const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
- err = txn->get(info_key, &info_val);
- if (err != TxnErrorCode::TXN_OK) {
- code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
- : cast_as<ErrCategory::READ>(err);
- if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
- } else {
- ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
- << " err=" << err;
+ // Get txn info with db_id and txn_id
+ std::string info_val; // Will be reused when saving updated txn
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
+ } else {
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
+ }
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(info_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // TODO: do more check like txn state, 2PC etc.
- DCHECK(txn_info.txn_id() == txn_id);
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
- code = MetaServiceCode::TXN_ALREADY_ABORTED;
- ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ // TODO: do more check like txn state, 2PC etc.
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
- if (request->has_is_2pc() && request->is_2pc()) {
- code = MetaServiceCode::TXN_ALREADY_VISIBLE;
- ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(INFO) << msg;
response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- code = MetaServiceCode::OK;
- ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(INFO) << msg;
- response->mutable_txn_info()->CopyFrom(txn_info);
- return;
- }
- if (request->has_is_2pc() && request->is_2pc() &&
- txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
- code = MetaServiceCode::TXN_INVALID_STATUS;
- ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Prepare rowset meta and new_versions
+ // Prepare rowset meta and new_versions
+ // Read tablet indexes in batch.
+ std::vector<std::string> tablet_idx_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
+ }
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ size_t total_rowsets = tmp_rowsets_meta.size();
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ for (size_t i = 0; i < total_rowsets; i++) {
+ uint64_t tablet_id = tmp_rowsets_meta[i].second.tablet_id();
+ if (!tablet_idx_values[i].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " tablet_id=" << tablet_id << " key=" << hex(tablet_idx_keys[i]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "malformed tablet index value tablet_id=" << tablet_id
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ VLOG_DEBUG << "tablet_id:" << tablet_id
+ << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ }
+
+ tablet_idx_keys.clear();
+ tablet_idx_values.clear();
+
+ // {table/partition} -> version
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ size_t total_versions = version_keys.size();
+ for (size_t i = 0; i < total_versions; i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+ version_keys.clear();
+ version_values.clear();
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
+ continue;
+ }
+
+ std::vector<std::pair<std::string, std::string>> rowsets;
+ std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
+ rowsets.reserve(tmp_rowsets_meta.size());
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
Review Comment:
do not change what you don't modify
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -862,94 +863,485 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
std::stringstream ss;
int64_t txn_id = request->txn_id();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // Get txn info with db_id and txn_id
- std::string info_val; // Will be reused when saving updated txn
- const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
- err = txn->get(info_key, &info_val);
- if (err != TxnErrorCode::TXN_OK) {
- code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
- : cast_as<ErrCategory::READ>(err);
- if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
- } else {
- ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
- << " err=" << err;
+ // Get txn info with db_id and txn_id
+ std::string info_val; // Will be reused when saving updated txn
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
+ } else {
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
+ }
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(info_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // TODO: do more check like txn state, 2PC etc.
- DCHECK(txn_info.txn_id() == txn_id);
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
- code = MetaServiceCode::TXN_ALREADY_ABORTED;
- ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ // TODO: do more check like txn state, 2PC etc.
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
- if (request->has_is_2pc() && request->is_2pc()) {
- code = MetaServiceCode::TXN_ALREADY_VISIBLE;
- ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(INFO) << msg;
response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- code = MetaServiceCode::OK;
- ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(INFO) << msg;
- response->mutable_txn_info()->CopyFrom(txn_info);
- return;
- }
- if (request->has_is_2pc() && request->is_2pc() &&
- txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
- code = MetaServiceCode::TXN_INVALID_STATUS;
- ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Prepare rowset meta and new_versions
+ // Prepare rowset meta and new_versions
+ // Read tablet indexes in batch.
+ std::vector<std::string> tablet_idx_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
+ }
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ size_t total_rowsets = tmp_rowsets_meta.size();
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ for (size_t i = 0; i < total_rowsets; i++) {
+ uint64_t tablet_id = tmp_rowsets_meta[i].second.tablet_id();
+ if (!tablet_idx_values[i].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " tablet_id=" << tablet_id << " key=" << hex(tablet_idx_keys[i]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "malformed tablet index value tablet_id=" << tablet_id
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ VLOG_DEBUG << "tablet_id:" << tablet_id
+ << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ }
+
+ tablet_idx_keys.clear();
+ tablet_idx_values.clear();
+
+ // {table/partition} -> version
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ size_t total_versions = version_keys.size();
+ for (size_t i = 0; i < total_versions; i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+ version_keys.clear();
+ version_values.clear();
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
+ continue;
+ }
+
+ std::vector<std::pair<std::string, std::string>> rowsets;
+ std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
+ rowsets.reserve(tmp_rowsets_meta.size());
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions[ver_key] == 0) [[unlikely]] {
+ // it is impossible.
+ code = MetaServiceCode::UNDEFINED_ERR;
+ ss << "failed to get partition version key, the target version not exists in "
+ "new_versions."
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(ERROR) << msg;
+ return;
+ }
+
+ // Update rowset version
+ int64_t new_version = new_versions[ver_key];
+ i.set_start_version(new_version);
+ i.set_end_version(new_version);
+
+ std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
+ std::string val;
+ if (!i.SerializeToString(&val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ rowsets.emplace_back(std::move(key), std::move(val));
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tablet_id];
+ stats.data_size += i.data_disk_size();
+ stats.num_rows += i.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += i.num_segments();
+ } // for tmp_rowsets_meta
+
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ size_t total_locks = lock_keys.size();
+ for (size_t i = 0; i < total_locks; i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
+
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
+
+ // Save rowset meta
+ for (auto& i : rowsets) {
+ size_t rowset_size = i.first.size() + i.second.size();
+ txn->put(i.first, i.second);
+ LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_size;
+ }
+
+ // Save versions
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
+
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
+
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
Review Comment:
do not change what you don't modify
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -862,94 +863,485 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
std::stringstream ss;
int64_t txn_id = request->txn_id();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // Get txn info with db_id and txn_id
- std::string info_val; // Will be reused when saving updated txn
- const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
- err = txn->get(info_key, &info_val);
- if (err != TxnErrorCode::TXN_OK) {
- code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
- : cast_as<ErrCategory::READ>(err);
- if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
- } else {
- ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
- << " err=" << err;
+ // Get txn info with db_id and txn_id
+ std::string info_val; // Will be reused when saving updated txn
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
+ } else {
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
+ }
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(info_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // TODO: do more check like txn state, 2PC etc.
- DCHECK(txn_info.txn_id() == txn_id);
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
- code = MetaServiceCode::TXN_ALREADY_ABORTED;
- ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ // TODO: do more check like txn state, 2PC etc.
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
- if (request->has_is_2pc() && request->is_2pc()) {
- code = MetaServiceCode::TXN_ALREADY_VISIBLE;
- ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(INFO) << msg;
response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- code = MetaServiceCode::OK;
- ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(INFO) << msg;
- response->mutable_txn_info()->CopyFrom(txn_info);
- return;
- }
- if (request->has_is_2pc() && request->is_2pc() &&
- txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
- code = MetaServiceCode::TXN_INVALID_STATUS;
- ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Prepare rowset meta and new_versions
+ // Prepare rowset meta and new_versions
+ // Read tablet indexes in batch.
+ std::vector<std::string> tablet_idx_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
+ }
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ size_t total_rowsets = tmp_rowsets_meta.size();
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ for (size_t i = 0; i < total_rowsets; i++) {
+ uint64_t tablet_id = tmp_rowsets_meta[i].second.tablet_id();
+ if (!tablet_idx_values[i].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " tablet_id=" << tablet_id << " key=" << hex(tablet_idx_keys[i]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "malformed tablet index value tablet_id=" << tablet_id
Review Comment:
do not change what you don't modify
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -862,94 +863,485 @@ void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stat
*/
void commit_txn_immediately(
const CommitTxnRequest* request, CommitTxnResponse* response,
- std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
- const std::string& instance_id, int64_t db_id,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
std::stringstream ss;
int64_t txn_id = request->txn_id();
- std::unique_ptr<Transaction> txn;
- TxnErrorCode err = txn_kv->create_txn(&txn);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::CREATE>(err);
- ss << "filed to create txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+ do {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // Get txn info with db_id and txn_id
- std::string info_val; // Will be reused when saving updated txn
- const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
- err = txn->get(info_key, &info_val);
- if (err != TxnErrorCode::TXN_OK) {
- code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
- : cast_as<ErrCategory::READ>(err);
- if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
- ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
- } else {
- ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
- << " err=" << err;
+ // Get txn info with db_id and txn_id
+ std::string info_val; // Will be reused when saving updated txn
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ ss << "transaction [" << txn_id << "] not found, db_id=" << db_id;
+ } else {
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
+ }
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
- TxnInfoPB txn_info;
- if (!txn_info.ParseFromString(info_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- // TODO: do more check like txn state, 2PC etc.
- DCHECK(txn_info.txn_id() == txn_id);
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
- code = MetaServiceCode::TXN_ALREADY_ABORTED;
- ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ // TODO: do more check like txn state, 2PC etc.
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction [" << txn_id << "] is already aborted, db_id=" << db_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
- if (request->has_is_2pc() && request->is_2pc()) {
- code = MetaServiceCode::TXN_ALREADY_VISIBLE;
- ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(INFO) << msg;
response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- code = MetaServiceCode::OK;
- ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
- msg = ss.str();
- LOG(INFO) << msg;
- response->mutable_txn_info()->CopyFrom(txn_info);
- return;
- }
- if (request->has_is_2pc() && request->is_2pc() &&
- txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
- code = MetaServiceCode::TXN_INVALID_STATUS;
- ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id" << txn_id;
- msg = ss.str();
- LOG(WARNING) << msg;
- return;
- }
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
- LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Prepare rowset meta and new_versions
+ // Prepare rowset meta and new_versions
+ // Read tablet indexes in batch.
+ std::vector<std::string> tablet_idx_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
+ }
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ size_t total_rowsets = tmp_rowsets_meta.size();
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ for (size_t i = 0; i < total_rowsets; i++) {
+ uint64_t tablet_id = tmp_rowsets_meta[i].second.tablet_id();
+ if (!tablet_idx_values[i].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " tablet_id=" << tablet_id << " key=" << hex(tablet_idx_keys[i]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "malformed tablet index value tablet_id=" << tablet_id
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ VLOG_DEBUG << "tablet_id:" << tablet_id
+ << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ }
+
+ tablet_idx_keys.clear();
+ tablet_idx_values.clear();
+
+ // {table/partition} -> version
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
+ }
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ size_t total_versions = version_keys.size();
+ for (size_t i = 0; i < total_versions; i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+ version_keys.clear();
+ version_values.clear();
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
+ continue;
+ }
+
+ std::vector<std::pair<std::string, std::string>> rowsets;
+ std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
+ rowsets.reserve(tmp_rowsets_meta.size());
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions[ver_key] == 0) [[unlikely]] {
+ // it is impossible.
+ code = MetaServiceCode::UNDEFINED_ERR;
+ ss << "failed to get partition version key, the target version not exists in "
+ "new_versions."
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(ERROR) << msg;
+ return;
+ }
+
+ // Update rowset version
+ int64_t new_version = new_versions[ver_key];
+ i.set_start_version(new_version);
+ i.set_end_version(new_version);
+
+ std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
+ std::string val;
+ if (!i.SerializeToString(&val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ rowsets.emplace_back(std::move(key), std::move(val));
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tablet_id];
+ stats.data_size += i.data_disk_size();
+ stats.num_rows += i.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += i.num_segments();
+ } // for tmp_rowsets_meta
+
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ size_t total_locks = lock_keys.size();
+ for (size_t i = 0; i < total_locks; i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
+
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
+
+ // Save rowset meta
+ for (auto& i : rowsets) {
+ size_t rowset_size = i.first.size() + i.second.size();
+ txn->put(i.first, i.second);
+ LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_size;
+ }
+
+ // Save versions
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
+
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
+
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
+ << " txn_id=" << txn_id;
+ }
+
+ LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
+
+ // Update txn_info
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ // Update stats of affected tablet
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+ // Remove tmp rowset meta
+ for (auto& [k, _] : tmp_rowsets_meta) {
+ txn->remove(k);
+ LOG(INFO) << "xxx remove tmp_rowset_key=" << hex(k) << " txn_id=" << txn_id;
+ }
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ recycle_pb.set_creation_time(commit_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
+ }
+
+ LOG(INFO) << "xxx commit_txn put recycle_key key=" << hex(recycle_key)
+ << " txn_id=" << txn_id;
+ LOG(INFO) << "commit_txn put_size=" << txn->put_bytes()
Review Comment:
do not change what you don't modify
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
Review Comment:
we need to check `if (!tablet_idx_pb.has_db_id())` before copying so many keys.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
Review Comment:
why not submit multiple pending txn_ids?
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
Review Comment:
`tablet_idx_keys` should be `sub_tablet_idx_keys`?
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
Review Comment:
LOG every key that is modified and its vaule too:
db_id tablet_id txn_id. it is affordabale because number of tablets finite andn this process is convergent.
and LOG some summary of these update: how many tablet_idx has been updated, txn_id.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
Review Comment:
keep `last_pending_txn_id` is enough, need_advance_last_txn is redundant.
and it seems that there may be more than one `last_pending_txn_id`s, becase each partition may have a pending txn attached to it.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
Review Comment:
add comment what the KV stands for
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
Review Comment:
add a summary here how long do we take before we reach here (there may be mutiple `task->wait()`s which may take a lot of time.)
LOG how may txn "advanced" and how long it takes + txn_id.
also make it a bvar to stats
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
Review Comment:
LOG txn id
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
Review Comment:
ditto
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
Review Comment:
dont log "xxx" any more...
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
Review Comment:
LOG right after parsing it from string at line 1601
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
Review Comment:
don't log "xxx"
##########
cloud/src/meta-service/txn_lazy_committer.h:
##########
Review Comment:
add comment for all structures and public functions
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ int retry_times = 0;
+ do {
+ LOG(INFO) << "lazy task commit txn_id=" << txn_id_ << " retry_times=" << retry_times;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end =
+ (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id_, txn_id_, txn_kv_, code_, msg_, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code_ != MetaServiceCode::OK) break;
+ }
+ if (code_ != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id_, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code_ = err == TxnErrorCode::TXN_KEY_NOT_FOUND
+ ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id_
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id_
+ << " key=" << hex(ver_key);
+ msg_ = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id_) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id_;
+ msg_ = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ break;
+ }
+ }
+ }
+ make_committed_txn_visible(instance_id_, db_id, txn_id_, txn_kv_, code_, msg_);
+ } while (false);
+ } while (code_ == MetaServiceCode::KV_TXN_CONFLICT &&
+ retry_times++ < config::txn_store_retry_times);
+ {
+ std::unique_lock<std::mutex> lock(mutex_);
+ this->finished_ = true;
+ }
+ this->cond_.notify_all();
+}
+
+std::pair<MetaServiceCode, std::string> TxnLazyCommitTask::wait() {
+ {
+ std::unique_lock<std::mutex> lock(mutex_);
+ cond_.wait(lock, [this]() { return this->finished_ == true; });
Review Comment:
wait with timeout with a loop, and LOG what is going on, how long does it take while waiting to be done.
and we may need to count how many `wait()` has been called for this pending txn.
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ int retry_times = 0;
+ do {
+ LOG(INFO) << "lazy task commit txn_id=" << txn_id_ << " retry_times=" << retry_times;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end =
+ (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id_, txn_id_, txn_kv_, code_, msg_, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code_ != MetaServiceCode::OK) break;
+ }
+ if (code_ != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id_, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code_ = err == TxnErrorCode::TXN_KEY_NOT_FOUND
+ ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id_
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id_
+ << " key=" << hex(ver_key);
+ msg_ = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id_) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id_;
+ msg_ = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ break;
+ }
+ }
+ }
+ make_committed_txn_visible(instance_id_, db_id, txn_id_, txn_kv_, code_, msg_);
+ } while (false);
+ } while (code_ == MetaServiceCode::KV_TXN_CONFLICT &&
+ retry_times++ < config::txn_store_retry_times);
+ {
+ std::unique_lock<std::mutex> lock(mutex_);
+ this->finished_ = true;
Review Comment:
do we count it as a finish even if `retry_times > config::txn_store_retry_times`? line 413 may be false and make it reach here.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
Review Comment:
add UT for this function to do some boundary tests of the "for loop"
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
+ if (ret != 0) [[unlikely]] {
+ // decode version key error means this is something wrong,
+ // we can not continue this txn
+ LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = "decode version key error";
+ return;
+ }
- // Save table versions
- for (auto& i : table_id_tablet_ids) {
- std::string ver_key = table_version_key({instance_id, db_id, i.first});
- txn->atomic_add(ver_key, 1);
- LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key) << " txn_id=" << txn_id;
- }
+ int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
+ int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
+ VLOG_DEBUG << "txn_id=" << txn_id << " table_id=" << table_id
+ << " partition_id=" << partition_id << " version=" << i.second;
- LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
+ response->add_table_ids(table_id);
+ response->add_partition_ids(partition_id);
+ response->add_versions(i.second);
+ }
- // Update txn_info
- txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+ // process mow table, check lock and remove pending key
+ std::vector<std::string> lock_keys;
+ lock_keys.reserve(request->mow_table_ids().size());
+ for (auto table_id : request->mow_table_ids()) {
+ lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
+ }
+ std::vector<std::optional<std::string>> lock_values;
+ err = txn->batch_get(&lock_values, lock_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
+ << " err=" << err;
+ msg = ss.str();
+ code = cast_as<ErrCategory::READ>(err);
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- auto now_time = system_clock::now();
- uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
- if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
- LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
- << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
- return;
- }
- txn_info.set_commit_time(commit_time);
- txn_info.set_finish_time(commit_time);
- if (request->has_commit_attachment()) {
- txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
- }
- LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
- info_val.clear();
- if (!txn_info.SerializeToString(&info_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(info_key, info_val);
- LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+ for (size_t i = 0; i < lock_keys.size(); i++) {
+ int64_t table_id = request->mow_table_ids(i);
+ // When the key does not exist, it means the lock has been acquired
+ // by another transaction and successfully committed.
+ if (!lock_values[i].has_value()) {
+ ss << "get delete bitmap update lock info, lock is expired"
+ << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
+ code = MetaServiceCode::LOCK_EXPIRED;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
- // Update stats of affected tablet
- for (auto& [tablet_id, stats] : tablet_stats) {
- DCHECK(tablet_ids.count(tablet_id));
- auto& tablet_idx = tablet_ids[tablet_id];
- StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
- tablet_idx.partition_id(), tablet_id};
- update_tablet_stats(info, stats, txn, code, msg);
- if (code != MetaServiceCode::OK) return;
- }
- // Remove tmp rowset meta
- for (auto& [k, _] : tmp_rowsets_meta) {
- txn->remove(k);
- LOG(INFO) << "xxx remove tmp_rowset_key=" << hex(k) << " txn_id=" << txn_id;
- }
+ DeleteBitmapUpdateLockPB lock_info;
+ if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "failed to parse DeleteBitmapUpdateLockPB";
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+ if (lock_info.lock_id() != request->txn_id()) {
+ msg = "lock is expired";
+ code = MetaServiceCode::LOCK_EXPIRED;
+ return;
+ }
+ txn->remove(lock_keys[i]);
+ LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
+ << " txn_id=" << txn_id;
- const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
- LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
- txn->remove(running_key);
+ for (auto tablet_id : table_id_tablet_ids[table_id]) {
+ std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
+ txn->remove(pending_key);
+ LOG(INFO) << "xxx remove delete bitmap pending key, pending_key="
+ << hex(pending_key) << " txn_id=" << txn_id;
+ }
+ }
+ lock_keys.clear();
+ lock_values.clear();
- std::string recycle_val;
- std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
- RecycleTxnPB recycle_pb;
- recycle_pb.set_creation_time(commit_time);
- recycle_pb.set_label(txn_info.label());
+ // Save table versions
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key)
+ << " txn_id=" << txn_id;
+ }
- if (!recycle_pb.SerializeToString(&recycle_val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
- msg = ss.str();
- return;
- }
- txn->put(recycle_key, recycle_val);
+ LOG(INFO) << "commit_txn_eventually put_size=" << txn->put_bytes()
+ << " del_size=" << txn->delete_bytes() << " num_put_keys=" << txn->num_put_keys()
+ << " num_del_keys=" << txn->num_del_keys()
+ << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
- if (txn_info.load_job_source_type() ==
- LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
- put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
- }
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
- LOG(INFO) << "xxx commit_txn put recycle_key key=" << hex(recycle_key) << " txn_id=" << txn_id;
- LOG(INFO) << "commit_txn put_size=" << txn->put_bytes() << " del_size=" << txn->delete_bytes()
- << " num_put_keys=" << txn->num_put_keys() << " num_del_keys=" << txn->num_del_keys()
- << " txn_size=" << txn->approximate_bytes() << " txn_id=" << txn_id;
+ std::shared_ptr<TxnLazyCommitTask> task = txn_lazy_committer->submit(instance_id, txn_id);
+ std::pair<MetaServiceCode, std::string> ret = task->wait();
+ if (ret.first != MetaServiceCode::OK) {
+ LOG(WARNING) << "lazy commit txn failed txn_id=" << txn_id << " code=" << ret.first
+ << "msg=" << ret.second;
+ }
- // Finally we are done...
- err = txn->commit();
- if (err != TxnErrorCode::TXN_OK) {
- if ((err == TxnErrorCode::TXN_VALUE_TOO_LARGE ||
- err == TxnErrorCode::TXN_BYTES_TOO_LARGE) &&
- !tmp_rowsets_meta.empty()) {
- size_t max_size = 0, max_idx = 0, max_num_segments = 0,
- min_num_segments = std::numeric_limits<size_t>::max(), avg_num_segments = 0;
- for (size_t i = 0; i < tmp_rowsets_meta.size(); i++) {
- auto& [k, v] = tmp_rowsets_meta[i];
- if (v.ByteSizeLong() > max_size) {
- max_size = v.ByteSizeLong();
- max_idx = i;
- }
- if (v.num_segments() > max_num_segments) {
- max_num_segments = v.num_segments();
- }
- if (v.num_segments() < min_num_segments) {
- min_num_segments = v.num_segments();
- }
- avg_num_segments += v.num_segments();
- }
- if (!tmp_rowsets_meta.empty()) {
- avg_num_segments /= tmp_rowsets_meta.size();
- }
- LOG(WARNING) << "failed to commit kv txn"
- << ", err=" << err << ", txn_id=" << txn_id
- << ", total_rowsets=" << tmp_rowsets_meta.size()
- << ", avg_num_segments=" << avg_num_segments
- << ", min_num_segments=" << min_num_segments
- << ", max_num_segments=" << max_num_segments
- << ", largest_rowset_size=" << max_size
- << ", largest_rowset_key=" << hex(tmp_rowsets_meta[max_idx].first)
- << ", largest_rowset_value="
- << tmp_rowsets_meta[max_idx].second.ShortDebugString();
+ std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ // Accumulate affected rows
+ auto& stats = tablet_stats[i.tablet_id()];
+ stats.data_size += i.data_disk_size();
+ stats.num_rows += i.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += i.num_segments();
}
- code = cast_as<ErrCategory::COMMIT>(err);
- ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
- msg = ss.str();
- return;
- }
- // calculate table stats from tablets stats
- std::map<int64_t /*table_id*/, TableStats> table_stats;
- std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
- request->base_tablet_ids().end());
- calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
- for (const auto& pair : table_stats) {
- TableStatsPB* stats_pb = response->add_table_stats();
- auto table_id = pair.first;
- auto stats = pair.second;
- get_pb_from_tablestats(stats, stats_pb);
- stats_pb->set_table_id(table_id);
- VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
- << " table_id=" << table_id
- << " updated_row_count=" << stats_pb->updated_row_count();
- }
+ // calculate table stats from tablets stats
+ std::map<int64_t /*table_id*/, TableStats> table_stats;
+ std::vector<int64_t> base_tablet_ids(request->base_tablet_ids().begin(),
+ request->base_tablet_ids().end());
+ calc_table_stats(tablet_ids, tablet_stats, table_stats, base_tablet_ids);
+ for (const auto& pair : table_stats) {
+ TableStatsPB* stats_pb = response->add_table_stats();
+ auto table_id = pair.first;
+ auto stats = pair.second;
+ get_pb_from_tablestats(stats, stats_pb);
+ stats_pb->set_table_id(table_id);
+ VLOG_DEBUG << "Add TableStats to CommitTxnResponse. txn_id=" << txn_id
+ << " table_id=" << table_id
+ << " updated_row_count=" << stats_pb->updated_row_count();
+ }
- response->mutable_txn_info()->CopyFrom(txn_info);
-} // end commit_txn
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
Review Comment:
is it VISIBLE even if `task->wait()` failed?
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
Review Comment:
is it a unrecoverable error?
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
Review Comment:
log
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
Review Comment:
key or partition id should be printed.
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ int retry_times = 0;
+ do {
+ LOG(INFO) << "lazy task commit txn_id=" << txn_id_ << " retry_times=" << retry_times;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
Review Comment:
add comment for the map, what is `std::pair<std::string, doris::RowsetMetaCloudPB>`
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
Review Comment:
and it should be a snapshot GET, concurrent updating does not matter, otherwise the update procedure may no succeed if there are other reads on tablet_idx.
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ int retry_times = 0;
+ do {
+ LOG(INFO) << "lazy task commit txn_id=" << txn_id_ << " retry_times=" << retry_times;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end =
+ (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id_, txn_id_, txn_kv_, code_, msg_, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code_ != MetaServiceCode::OK) break;
+ }
+ if (code_ != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
Review Comment:
LOG every key removed.
##########
cloud/src/meta-service/meta_service_txn.cpp:
##########
@@ -974,353 +1361,476 @@ void commit_txn_immediately(
LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
return;
}
- if (!tablet_ids[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
+ if (!(*tablet_ids)[tablet_id].ParseFromString(tablet_idx_values[i].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
ss << "malformed tablet index value tablet_id=" << tablet_id << " txn_id=" << txn_id;
msg = ss.str();
LOG(WARNING) << msg;
return;
}
- table_id_tablet_ids[tablet_ids[tablet_id].table_id()].push_back(tablet_id);
+ if (!(*tablet_ids)[tablet_id].has_db_id()) {
+ *need_repair_tablet_idx = true;
+ }
+ (*table_id_tablet_ids)[(*tablet_ids)[tablet_id].table_id()].push_back(tablet_id);
VLOG_DEBUG << "tablet_id:" << tablet_id
- << " value:" << tablet_ids[tablet_id].ShortDebugString();
+ << " value:" << (*tablet_ids)[tablet_id].ShortDebugString();
}
tablet_idx_keys.clear();
tablet_idx_values.clear();
+}
- // {table/partition} -> version
- std::unordered_map<std::string, uint64_t> new_versions;
- std::vector<std::string> version_keys;
+void repair_tablet_index(
+ std::shared_ptr<TxnKv>& txn_kv, MetaServiceCode& code, std::string& msg,
+ const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ std::vector<std::string> tablet_idx_keys;
for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions.count(ver_key) == 0) {
- new_versions.insert({ver_key, 0});
- version_keys.push_back(std::move(ver_key));
- }
- }
- std::vector<std::optional<std::string>> version_values;
- err = txn->batch_get(&version_values, version_keys);
- if (err != TxnErrorCode::TXN_OK) {
- code = cast_as<ErrCategory::READ>(err);
- ss << "failed to get partition versions, err=" << err;
- msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
+ tablet_idx_keys.push_back(meta_tablet_idx_key({instance_id, i.tablet_id()}));
}
- size_t total_versions = version_keys.size();
- for (size_t i = 0; i < total_versions; i++) {
- int64_t version;
- if (version_values[i].has_value()) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(version_values[i].value())) {
+
+#define MAX_TABLET_INDEX_NUM 2048
+ for (size_t i = 0; i < tablet_idx_keys.size(); i += MAX_TABLET_INDEX_NUM) {
+ size_t end = (i + MAX_TABLET_INDEX_NUM) > tablet_idx_keys.size() ? tablet_idx_keys.size()
+ : i + MAX_TABLET_INDEX_NUM;
+ const std::vector<std::string> sub_tablet_idx_keys(tablet_idx_keys.begin() + i,
+ tablet_idx_keys.begin() + end);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::vector<std::optional<std::string>> tablet_idx_values;
+ // batch get snapshot is false
+ err = txn->batch_get(&tablet_idx_values, tablet_idx_keys,
+ Transaction::BatchGetOptions(false));
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet table index ids, err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
+ return;
+ }
+
+ for (size_t j = 0; j < sub_tablet_idx_keys.size(); j++) {
+ if (!tablet_idx_values[j].has_value()) [[unlikely]] {
+ // The value must existed
+ code = MetaServiceCode::KV_TXN_GET_ERR;
+ ss << "failed to get tablet table index ids, err=not found"
+ << " key=" << hex(tablet_idx_keys[j]);
+ msg = ss.str();
+ LOG(WARNING) << msg << " err=" << err << " txn_id=" << txn_id;
+ return;
+ }
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_values[j].value())) [[unlikely]] {
code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- ss << "failed to parse version pb txn_id=" << txn_id
- << " key=" << hex(version_keys[i]);
+ ss << "malformed tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- version = version_pb.version();
- } else {
- version = 1;
+
+ if (!tablet_idx_pb.has_db_id()) {
+ tablet_idx_pb.set_db_id(db_id);
+ std::string idx_val;
+ if (!tablet_idx_pb.SerializeToString(&idx_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize tablet index value key=" << hex(tablet_idx_keys[j])
+ << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ txn->put(sub_tablet_idx_keys[j], idx_val);
+ }
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ }
+}
+
+void commit_txn_eventually(
+ const CommitTxnRequest* request, CommitTxnResponse* response,
+ std::shared_ptr<TxnKv>& txn_kv, std::shared_ptr<TxnLazyCommitter>& txn_lazy_committer,
+ MetaServiceCode& code, std::string& msg, const std::string& instance_id, int64_t db_id,
+ const std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta) {
+ std::stringstream ss;
+ TxnErrorCode err = TxnErrorCode::TXN_OK;
+ int64_t txn_id = request->txn_id();
+ bool need_advance_last_txn = false;
+ int64_t last_pending_txn_id = 0;
+
+ do {
+ std::unique_ptr<Transaction> txn;
+ err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // tablet_id -> {table/index/partition}_id
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ // table_id -> tablets_ids
+ std::unordered_map<int64_t, std::vector<int64_t>> table_id_tablet_ids;
+ bool need_repair_tablet_idx = false;
+ get_tablet_indexes(instance_id, txn_id, tmp_rowsets_meta, txn, code, msg, &tablet_ids,
+ &table_id_tablet_ids, &need_repair_tablet_idx);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "get_tablet_indexes failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+
+ if (need_repair_tablet_idx) {
+ txn.reset();
+ repair_tablet_index(txn_kv, code, msg, instance_id, db_id, txn_id, tmp_rowsets_meta);
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "repair_tablet_index failed, txn_id=" << txn_id << " code=" << code;
+ return;
+ }
+ continue;
+ }
+
+ std::unordered_map<std::string, uint64_t> new_versions;
+ std::vector<std::string> version_keys;
+ for (auto& [_, i] : tmp_rowsets_meta) {
+ int64_t tablet_id = i.tablet_id();
+ int64_t table_id = tablet_ids[tablet_id].table_id();
+ int64_t partition_id = i.partition_id();
+ std::string ver_key =
+ partition_version_key({instance_id, db_id, table_id, partition_id});
+ if (new_versions.count(ver_key) == 0) {
+ new_versions.insert({ver_key, 0});
+ version_keys.push_back(std::move(ver_key));
+ }
}
- new_versions[version_keys[i]] = version + 1;
- }
- version_keys.clear();
- version_values.clear();
- std::vector<std::pair<std::string, std::string>> rowsets;
- std::unordered_map<int64_t, TabletStats> tablet_stats; // tablet_id -> stats
- rowsets.reserve(tmp_rowsets_meta.size());
- for (auto& [_, i] : tmp_rowsets_meta) {
- int64_t tablet_id = i.tablet_id();
- int64_t table_id = tablet_ids[tablet_id].table_id();
- int64_t partition_id = i.partition_id();
- std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
- if (new_versions[ver_key] == 0) [[unlikely]] {
- // it is impossible.
- code = MetaServiceCode::UNDEFINED_ERR;
- ss << "failed to get partition version key, the target version not exists in "
- "new_versions."
- << " txn_id=" << txn_id;
+ std::vector<std::optional<std::string>> version_values;
+ err = txn->batch_get(&version_values, version_keys);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partition versions, err=" << err;
msg = ss.str();
- LOG(ERROR) << msg;
+ LOG(WARNING) << msg << " txn_id=" << txn_id;
return;
}
- // Update rowset version
- int64_t new_version = new_versions[ver_key];
- i.set_start_version(new_version);
- i.set_end_version(new_version);
+ for (size_t i = 0; i < version_keys.size(); i++) {
+ int64_t version;
+ if (version_values[i].has_value()) {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(version_values[i].value())) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id
+ << " key=" << hex(version_keys[i]);
+ msg = ss.str();
+ return;
+ }
+ if (version_pb.has_txn_id()) {
+ need_advance_last_txn = true;
+ last_pending_txn_id = version_pb.txn_id();
+ break;
+ }
+ version = version_pb.version();
+ } else {
+ version = 1;
+ }
+ new_versions[version_keys[i]] = version + 1;
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ }
+
+ if (need_advance_last_txn) {
+ txn.reset();
+ DCHECK(last_pending_txn_id > 0);
+ std::shared_ptr<TxnLazyCommitTask> task =
+ txn_lazy_committer->submit(instance_id, last_pending_txn_id);
+
+ std::tie(code, msg) = task->wait();
+ need_advance_last_txn = false;
+ last_pending_txn_id = 0;
+ if (code != MetaServiceCode::OK) {
+ LOG(WARNING) << "advance_last_txn failed last_txn=" << last_pending_txn_id
+ << " code=" << code << "msg=" << msg;
+ return;
+ }
- std::string key = meta_rowset_key({instance_id, tablet_id, i.end_version()});
- std::string val;
- if (!i.SerializeToString(&val)) {
- code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize rowset_meta, txn_id=" << txn_id;
+ // there maybe concurrent commit_txn_eventually, so we need continue to make sure
+ // partition versionPB has no txn_id
+ continue;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id
+ << " err=" << err;
msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- rowsets.emplace_back(std::move(key), std::move(val));
-
- // Accumulate affected rows
- auto& stats = tablet_stats[tablet_id];
- stats.data_size += i.data_disk_size();
- stats.num_rows += i.num_rows();
- ++stats.num_rowsets;
- stats.num_segs += i.num_segments();
- } // for tmp_rowsets_meta
- // process mow table, check lock and remove pending key
- std::vector<std::string> lock_keys;
- lock_keys.reserve(request->mow_table_ids().size());
- for (auto table_id : request->mow_table_ids()) {
- lock_keys.push_back(meta_delete_bitmap_update_lock_key({instance_id, table_id, -1}));
- }
- std::vector<std::optional<std::string>> lock_values;
- err = txn->batch_get(&lock_values, lock_keys);
- if (err != TxnErrorCode::TXN_OK) {
- ss << "failed to get delete bitmap update lock key info, instance_id=" << instance_id
- << " err=" << err;
- msg = ss.str();
- code = cast_as<ErrCategory::READ>(err);
- LOG(WARNING) << msg << " txn_id=" << txn_id;
- return;
- }
- size_t total_locks = lock_keys.size();
- for (size_t i = 0; i < total_locks; i++) {
- int64_t table_id = request->mow_table_ids(i);
- // When the key does not exist, it means the lock has been acquired
- // by another transaction and successfully committed.
- if (!lock_values[i].has_value()) {
- ss << "get delete bitmap update lock info, lock is expired"
- << " table_id=" << table_id << " key=" << hex(lock_keys[i]);
- code = MetaServiceCode::LOCK_EXPIRED;
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, db_id=" << db_id << " txn_id=" << txn_id;
msg = ss.str();
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ LOG(WARNING) << msg;
return;
}
- DeleteBitmapUpdateLockPB lock_info;
- if (!lock_info.ParseFromString(lock_values[i].value())) [[unlikely]] {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "failed to parse DeleteBitmapUpdateLockPB";
- LOG(WARNING) << msg << " txn_id=" << txn_id;
+ DCHECK(txn_info.txn_id() == txn_id);
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_ABORTED) {
+ code = MetaServiceCode::TXN_ALREADY_ABORTED;
+ ss << "transaction is already aborted: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
return;
}
- if (lock_info.lock_id() != request->txn_id()) {
- msg = "lock is expired";
- code = MetaServiceCode::LOCK_EXPIRED;
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE) {
+ if (request->has_is_2pc() && request->is_2pc()) {
+ code = MetaServiceCode::TXN_ALREADY_VISIBLE;
+ ss << "transaction [" << txn_id << "] is already visible, not pre-committed.";
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
+ return;
+ }
+ code = MetaServiceCode::OK;
+ ss << "transaction is already visible: db_id=" << db_id << " txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(INFO) << msg;
+ response->mutable_txn_info()->CopyFrom(txn_info);
return;
}
- txn->remove(lock_keys[i]);
- LOG(INFO) << "xxx remove delete bitmap lock, lock_key=" << hex(lock_keys[i])
- << " txn_id=" << txn_id;
- for (auto tablet_id : table_id_tablet_ids[table_id]) {
- std::string pending_key = meta_pending_delete_bitmap_key({instance_id, tablet_id});
- txn->remove(pending_key);
- LOG(INFO) << "xxx remove delete bitmap pending key, pending_key=" << hex(pending_key)
- << " txn_id=" << txn_id;
+ if (request->has_is_2pc() && request->is_2pc() &&
+ txn_info.status() == TxnStatusPB::TXN_STATUS_PREPARED) {
+ code = MetaServiceCode::TXN_INVALID_STATUS;
+ ss << "transaction is prepare, not pre-committed: db_id=" << db_id << " txn_id"
+ << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
}
- }
- lock_keys.clear();
- lock_values.clear();
+ LOG(INFO) << "txn_id=" << txn_id << " txn_info=" << txn_info.ShortDebugString();
- // Save rowset meta
- for (auto& i : rowsets) {
- size_t rowset_size = i.first.size() + i.second.size();
- txn->put(i.first, i.second);
- LOG(INFO) << "xxx put rowset_key=" << hex(i.first) << " txn_id=" << txn_id
- << " rowset_size=" << rowset_size;
- }
+ auto now_time = system_clock::now();
+ uint64_t commit_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ if ((txn_info.prepare_time() + txn_info.timeout_ms()) < commit_time) {
+ code = MetaServiceCode::UNDEFINED_ERR;
+ msg = fmt::format("txn is expired, not allow to commit txn_id={}", txn_id);
+ LOG(INFO) << msg << " prepare_time=" << txn_info.prepare_time()
+ << " timeout_ms=" << txn_info.timeout_ms() << " commit_time=" << commit_time;
+ return;
+ }
+ txn_info.set_commit_time(commit_time);
+ txn_info.set_finish_time(commit_time);
+ if (request->has_commit_attachment()) {
+ txn_info.mutable_commit_attachment()->CopyFrom(request->commit_attachment());
+ }
+ DCHECK(txn_info.status() != TxnStatusPB::TXN_STATUS_COMMITTED);
+ // lazy commit set status TXN_STATUS_COMMITTED not TXN_STATUS_VISIBLE !!!
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_COMMITTED);
- // Save versions
- int64_t version_update_time_ms =
- duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
- response->set_version_update_time_ms(version_update_time_ms);
- for (auto& i : new_versions) {
- std::string ver_val;
- VersionPB version_pb;
- version_pb.set_version(i.second);
- version_pb.set_update_time_ms(version_update_time_ms);
- if (!version_pb.SerializeToString(&ver_val)) {
+ LOG(INFO) << "after update txn_info=" << txn_info.ShortDebugString();
+ info_val.clear();
+ if (!txn_info.SerializeToString(&info_val)) {
code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
- ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
msg = ss.str();
return;
}
- txn->put(i.first, ver_val);
- LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
- << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
- std::string_view ver_key = i.first;
- ver_key.remove_prefix(1); // Remove key space
- // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
- std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
- int ret = decode_key(&ver_key, &out);
- if (ret != 0) [[unlikely]] {
- // decode version key error means this is something wrong,
- // we can not continue this txn
- LOG(WARNING) << "failed to decode key, ret=" << ret << " key=" << hex(ver_key);
- code = MetaServiceCode::UNDEFINED_ERR;
- msg = "decode version key error";
- return;
+ if (txn_info.load_job_source_type() ==
+ LoadJobSourceTypePB::LOAD_JOB_SRC_TYPE_ROUTINE_LOAD_TASK) {
+ put_routine_load_progress(code, msg, instance_id, request, txn.get(), db_id);
}
- int64_t table_id = std::get<int64_t>(std::get<0>(out[4]));
- int64_t partition_id = std::get<int64_t>(std::get<0>(out[5]));
- VLOG_DEBUG << " table_id=" << table_id << " partition_id=" << partition_id;
+ // save versions for partition
+ int64_t version_update_time_ms =
+ duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ response->set_version_update_time_ms(version_update_time_ms);
+ for (auto& i : new_versions) {
+ std::string ver_val;
+ VersionPB version_pb;
+ version_pb.set_version(i.second);
+ version_pb.set_txn_id(txn_id);
+ version_pb.set_update_time_ms(version_update_time_ms);
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
- response->add_table_ids(table_id);
- response->add_partition_ids(partition_id);
- response->add_versions(i.second);
- }
+ txn->put(i.first, ver_val);
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
+ << " txn_id=" << txn_id << " update_time=" << version_update_time_ms;
+
+ std::string_view ver_key = i.first;
+ ver_key.remove_prefix(1); // Remove key space
+ // PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
+ int ret = decode_key(&ver_key, &out);
Review Comment:
is this a redundant check?
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ int retry_times = 0;
+ do {
+ LOG(INFO) << "lazy task commit txn_id=" << txn_id_ << " retry_times=" << retry_times;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
Review Comment:
how come `tmp_rowset_metas.size() == 0` may exist?
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ DCHECK(version_pb.has_txn_id());
+ DCHECK(version_pb.txn_id() == txn_id);
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+
+ std::string rowset_key =
+ meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version_pb.version()});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version_pb.version());
+ tmp_rowset_pb.set_end_version(version_pb.version());
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "xxx put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "xxx put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "xxx remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(recycle_key, recycle_val);
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
+ std::stringstream ss;
+ int retry_times = 0;
+ do {
+ LOG(INFO) << "lazy task commit txn_id=" << txn_id_ << " retry_times=" << retry_times;
+ do {
+ code_ = MetaServiceCode::OK;
+ msg_.clear();
+ int64_t db_id;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>> tmp_rowset_metas;
+ scan_tmp_rowset(instance_id_, txn_id_, txn_kv_, code_, msg_, &db_id, &tmp_rowset_metas);
+ if (code_ != MetaServiceCode::OK) {
+ LOG(WARNING) << "scan_tmp_rowset failed, txn_id=" << txn_id_ << " code=" << code_;
+ break;
+ }
+
+ VLOG_DEBUG << "txn_id=" << txn_id_
+ << " tmp_rowset_metas.size()=" << tmp_rowset_metas.size();
+ if (tmp_rowset_metas.size() == 0) {
+ LOG(INFO) << "empty tmp_rowset_metas, txn_id=" << txn_id_;
+ }
+
+ // <partition_id, tmp_rowsets>
+ std::unordered_map<int64_t,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>>
+ partition_to_tmp_rowset_metas;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].emplace_back();
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().first =
+ tmp_rowset_key;
+ partition_to_tmp_rowset_metas[tmp_rowset_pb.partition_id()].back().second =
+ tmp_rowset_pb;
+ }
+
+ // tablet_id -> TabletIndexPB
+ std::unordered_map<int64_t, TabletIndexPB> tablet_ids;
+ for (auto& [partition_id, tmp_rowset_metas] : partition_to_tmp_rowset_metas) {
+ for (size_t i = 0; i < tmp_rowset_metas.size();
+ i += config::txn_lazy_max_rowsets_per_batch) {
+ size_t end =
+ (i + config::txn_lazy_max_rowsets_per_batch) > tmp_rowset_metas.size()
+ ? tmp_rowset_metas.size()
+ : i + config::txn_lazy_max_rowsets_per_batch;
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>
+ sub_partition_tmp_rowset_metas(tmp_rowset_metas.begin() + i,
+ tmp_rowset_metas.begin() + end);
+ convert_tmp_rowsets(instance_id_, txn_id_, txn_kv_, code_, msg_, db_id,
+ sub_partition_tmp_rowset_metas, tablet_ids);
+ if (code_ != MetaServiceCode::OK) break;
+ }
+ if (code_ != MetaServiceCode::OK) break;
+
+ DCHECK(tmp_rowset_metas.size() > 0);
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv_->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code_ = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id_ << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+
+ int64_t table_id = -1;
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowset_metas) {
+ if (table_id <= 0) {
+ table_id = tablet_ids[tmp_rowset_pb.tablet_id()].table_id();
+ }
+ txn->remove(tmp_rowset_key);
+ }
+
+ DCHECK(table_id > 0);
+ DCHECK(partition_id > 0);
+
+ std::string ver_val;
+ std::string ver_key =
+ partition_version_key({instance_id_, db_id, table_id, partition_id});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code_ = err == TxnErrorCode::TXN_KEY_NOT_FOUND
+ ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id_
+ << " key=" << hex(ver_key) << " err=" << err;
+ msg_ = ss.str();
+ LOG(WARNING) << msg_;
+ break;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id_
+ << " key=" << hex(ver_key);
+ msg_ = ss.str();
+ break;
+ }
+
+ if (version_pb.has_txn_id() && version_pb.txn_id() == txn_id_) {
+ version_pb.clear_txn_id();
+ ver_val.clear();
+ if (!version_pb.SerializeToString(&ver_val)) {
+ code_ = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize version_pb when saving, txn_id=" << txn_id_;
+ msg_ = ss.str();
+ return;
+ }
+ txn->put(ver_key, ver_val);
Review Comment:
log every key put
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312250476
<details>
<summary>TPC-DS: <b>Total hot run time: 192630 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb825b12499087f8dea29592d393d5582d3623c3, data reload: false
query1 1267 889 893 889
query2 6247 1929 1946 1929
query3 10625 3993 3895 3895
query4 60205 27074 23094 23094
query5 5296 508 507 507
query6 412 170 168 168
query7 5763 294 299 294
query8 296 222 227 222
query9 8845 2501 2478 2478
query10 500 280 273 273
query11 15821 15065 15283 15065
query12 168 105 108 105
query13 1572 425 395 395
query14 11222 7009 7292 7009
query15 244 174 186 174
query16 7576 462 451 451
query17 1113 589 558 558
query18 1933 299 296 296
query19 286 151 144 144
query20 122 117 114 114
query21 210 103 100 100
query22 4563 4365 4426 4365
query23 34185 33807 33264 33264
query24 5956 2909 2875 2875
query25 551 399 408 399
query26 692 163 165 163
query27 1782 290 295 290
query28 3815 2059 2050 2050
query29 707 429 434 429
query30 234 154 151 151
query31 945 752 761 752
query32 80 56 55 55
query33 426 296 291 291
query34 864 490 510 490
query35 826 728 704 704
query36 1079 918 938 918
query37 134 80 84 80
query38 4008 3948 3925 3925
query39 1450 1385 1382 1382
query40 203 121 121 121
query41 47 45 49 45
query42 118 98 96 96
query43 525 486 477 477
query44 1107 749 753 749
query45 198 167 177 167
query46 1085 753 775 753
query47 1907 1804 1823 1804
query48 381 293 299 293
query49 772 438 445 438
query50 829 436 418 418
query51 7265 7100 7038 7038
query52 103 94 88 88
query53 255 184 182 182
query54 579 464 465 464
query55 79 77 82 77
query56 287 269 269 269
query57 1207 1098 1069 1069
query58 238 252 233 233
query59 3167 2744 2774 2744
query60 306 293 316 293
query61 126 122 119 119
query62 751 646 646 646
query63 218 189 187 187
query64 2871 762 739 739
query65 3283 3149 3118 3118
query66 676 347 349 347
query67 15459 15115 15364 15115
query68 4459 574 548 548
query69 418 276 280 276
query70 1178 1067 1039 1039
query71 367 275 265 265
query72 6636 4016 4005 4005
query73 769 335 332 332
query74 9094 9154 8851 8851
query75 3411 2700 2697 2697
query76 1711 1062 1038 1038
query77 539 321 317 317
query78 9798 9320 9081 9081
query79 1541 534 534 534
query80 846 527 543 527
query81 570 230 222 222
query82 302 134 135 134
query83 186 145 155 145
query84 262 75 81 75
query85 885 290 291 290
query86 403 287 302 287
query87 4368 4263 4310 4263
query88 3894 2364 2353 2353
query89 390 294 292 292
query90 2000 202 194 194
query91 122 100 102 100
query92 61 53 54 53
query93 1686 538 538 538
query94 851 307 265 265
query95 356 259 275 259
query96 604 274 272 272
query97 3207 3091 3106 3091
query98 227 207 205 205
query99 1535 1250 1285 1250
Total cold run time: 308738 ms
Total hot run time: 192630 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2245562478
<details>
<summary>ClickBench: <b>Total hot run time: 30.15 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7e2b1bd85dd3827c86fb9ead1aa462b016a3aba, data reload: false
query1 0.04 0.03 0.04
query2 0.07 0.03 0.04
query3 0.22 0.05 0.05
query4 1.69 0.08 0.07
query5 0.49 0.49 0.48
query6 1.13 0.73 0.73
query7 0.02 0.02 0.01
query8 0.06 0.04 0.05
query9 0.56 0.50 0.47
query10 0.53 0.52 0.53
query11 0.16 0.12 0.12
query12 0.14 0.12 0.13
query13 0.59 0.58 0.58
query14 0.76 0.77 0.79
query15 0.86 0.82 0.81
query16 0.36 0.37 0.36
query17 0.96 1.04 1.00
query18 0.22 0.21 0.21
query19 1.75 1.65 1.81
query20 0.01 0.00 0.00
query21 15.40 0.72 0.65
query22 3.85 7.47 1.64
query23 18.32 1.43 1.29
query24 2.08 0.25 0.21
query25 0.16 0.08 0.08
query26 0.29 0.22 0.20
query27 0.45 0.23 0.23
query28 13.26 1.01 1.00
query29 12.59 3.30 3.29
query30 0.25 0.06 0.05
query31 2.86 0.39 0.40
query32 3.27 0.47 0.48
query33 2.88 2.87 2.89
query34 16.89 4.36 4.34
query35 4.42 4.35 4.38
query36 0.65 0.47 0.48
query37 0.19 0.15 0.16
query38 0.15 0.14 0.15
query39 0.04 0.03 0.04
query40 0.15 0.12 0.12
query41 0.10 0.04 0.05
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 108.98 s
Total hot run time: 30.15 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2244632951
<details>
<summary>TPC-H: <b>Total hot run time: 40396 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0cac9c57dfbabd3aa3c3d90819462ad7e839f881, data reload: false
------ Round 1 ----------------------------------
q1 18633 4997 4405 4405
q2 2629 208 199 199
q3 10973 1205 1193 1193
q4 10788 775 851 775
q5 7688 2793 2838 2793
q6 234 141 140 140
q7 976 616 618 616
q8 9274 2109 2125 2109
q9 8620 6579 6551 6551
q10 8858 3801 3825 3801
q11 458 236 241 236
q12 399 234 235 234
q13 17763 2955 2971 2955
q14 289 233 233 233
q15 545 484 505 484
q16 474 399 377 377
q17 992 672 679 672
q18 8226 7563 7372 7372
q19 4583 1469 1438 1438
q20 683 320 342 320
q21 5056 3211 4018 3211
q22 361 296 282 282
Total cold run time: 118502 ms
Total hot run time: 40396 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4353 4246 4222 4222
q2 388 268 254 254
q3 2983 2766 2793 2766
q4 1932 1641 1607 1607
q5 5302 5353 5316 5316
q6 220 134 135 134
q7 2144 1702 1742 1702
q8 3198 3337 3321 3321
q9 8446 8429 8428 8428
q10 3908 3728 3753 3728
q11 576 522 494 494
q12 778 592 613 592
q13 16554 2967 2973 2967
q14 295 273 260 260
q15 518 485 485 485
q16 491 425 417 417
q17 1782 1469 1452 1452
q18 7708 7624 7355 7355
q19 2722 1662 1488 1488
q20 1994 1794 1801 1794
q21 4898 4744 4760 4744
q22 589 485 508 485
Total cold run time: 71779 ms
Total hot run time: 54011 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2249779271
<details>
<summary>ClickBench: <b>Total hot run time: 30.88 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf6aad6e7d1b3229f6fdc1b41de15dc9b7d01bfc, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.22 0.06 0.05
query4 1.68 0.07 0.07
query5 0.48 0.49 0.49
query6 1.14 0.73 0.72
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.55 0.49 0.50
query10 0.56 0.55 0.55
query11 0.15 0.12 0.11
query12 0.15 0.13 0.12
query13 0.60 0.60 0.58
query14 0.77 0.78 0.79
query15 0.86 0.83 0.82
query16 0.36 0.36 0.36
query17 0.96 0.94 0.97
query18 0.22 0.22 0.21
query19 1.80 1.69 1.70
query20 0.01 0.01 0.02
query21 15.39 0.72 0.66
query22 4.09 6.84 2.18
query23 18.32 1.46 1.28
query24 2.07 0.22 0.21
query25 0.15 0.09 0.09
query26 0.30 0.22 0.21
query27 0.46 0.24 0.23
query28 13.28 1.03 0.99
query29 12.60 3.31 3.31
query30 0.26 0.06 0.05
query31 2.89 0.39 0.39
query32 3.25 0.47 0.46
query33 2.92 2.89 2.94
query34 17.08 4.35 4.33
query35 4.43 4.42 4.40
query36 0.66 0.51 0.48
query37 0.20 0.16 0.16
query38 0.17 0.15 0.17
query39 0.05 0.04 0.04
query40 0.15 0.12 0.13
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.06 0.04 0.04
Total cold run time: 109.64 s
Total hot run time: 30.88 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2249743123
<details>
<summary>TPC-H: <b>Total hot run time: 39710 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf6aad6e7d1b3229f6fdc1b41de15dc9b7d01bfc, data reload: false
------ Round 1 ----------------------------------
q1 18903 6335 4278 4278
q2 2011 188 197 188
q3 10567 1129 1077 1077
q4 10335 730 832 730
q5 7563 2711 2673 2673
q6 224 137 136 136
q7 950 597 584 584
q8 9216 2068 2049 2049
q9 8626 6526 6530 6526
q10 8716 3756 3758 3756
q11 483 238 245 238
q12 411 226 224 224
q13 17774 2982 2988 2982
q14 284 237 263 237
q15 522 486 493 486
q16 511 392 400 392
q17 986 696 695 695
q18 8084 7442 7301 7301
q19 4948 1442 1416 1416
q20 642 329 317 317
q21 4837 3154 3201 3154
q22 344 273 271 271
Total cold run time: 116937 ms
Total hot run time: 39710 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4377 4203 4201 4201
q2 360 268 262 262
q3 3047 2901 2956 2901
q4 1993 1654 1720 1654
q5 5585 5531 5489 5489
q6 220 145 129 129
q7 2223 1884 1828 1828
q8 3276 3399 3385 3385
q9 8726 8840 8816 8816
q10 4142 3798 3920 3798
q11 606 510 523 510
q12 811 659 657 657
q13 15727 3230 3188 3188
q14 322 291 295 291
q15 556 484 503 484
q16 500 440 422 422
q17 1845 1550 1520 1520
q18 8033 7955 7709 7709
q19 1778 1544 1534 1534
q20 2240 1880 1879 1879
q21 8261 4891 4857 4857
q22 609 541 519 519
Total cold run time: 75237 ms
Total hot run time: 56033 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2314365746
<details>
<summary>TPC-H: <b>Total hot run time: 38261 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cbbeda91329c7f9ecaf9f55ca0b98af79b86d7ee, data reload: false
------ Round 1 ----------------------------------
q1 18891 5287 4355 4355
q2 2054 189 189 189
q3 11442 964 1144 964
q4 10333 693 732 693
q5 7762 2882 2904 2882
q6 228 140 137 137
q7 974 617 601 601
q8 9327 2133 2112 2112
q9 7352 6556 6576 6556
q10 7006 2323 2297 2297
q11 441 246 245 245
q12 393 224 230 224
q13 17765 3030 3037 3030
q14 274 240 242 240
q15 528 469 488 469
q16 586 527 503 503
q17 993 694 670 670
q18 7533 6842 6862 6842
q19 1386 1016 1088 1016
q20 696 341 326 326
q21 4552 2969 2888 2888
q22 1152 1032 1022 1022
Total cold run time: 111668 ms
Total hot run time: 38261 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4435 4335 4320 4320
q2 383 271 267 267
q3 2892 2634 2744 2634
q4 1934 1718 1734 1718
q5 5725 5694 5837 5694
q6 225 133 133 133
q7 2202 1793 1847 1793
q8 3329 3488 3533 3488
q9 8904 8917 8871 8871
q10 3577 3438 3414 3414
q11 588 516 501 501
q12 865 677 678 677
q13 13058 3068 3223 3068
q14 320 278 291 278
q15 531 508 488 488
q16 648 576 602 576
q17 1851 1567 1557 1557
q18 8236 7798 7811 7798
q19 1745 1630 1537 1537
q20 2196 1923 1906 1906
q21 5786 5462 5503 5462
q22 1149 1037 1035 1035
Total cold run time: 70579 ms
Total hot run time: 57215 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1733771142
##########
cloud/src/recycler/recycler.cpp:
##########
@@ -1222,6 +1309,10 @@ int InstanceRecycler::recycle_tablets(int64_t table_id, int64_t index_id, int64_
int ret = scan_and_recycle(tablet_key_begin, tablet_key_end, std::move(recycle_func),
std::move(loop_done));
+ if (ret != 0) {
+ LOG(WARNING) << "failed to scan_and_recycle, instance_id=" << instance_id_;
Review Comment:
in the internal call `func`, it has printed more detail log when `ret != 0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1733937630
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,476 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include <chrono>
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+#include "meta-service/meta_service_tablet_stats.h"
+
+using namespace std::chrono;
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void update_tablet_stats(const StatsTabletKeyInfo& info, const TabletStats& stats,
+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids) {
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ // partition_id -> VersionPB
+ std::unordered_map<int64_t, VersionPB> partition_versions;
+ // tablet_id -> stats
+ std::unordered_map<int64_t, TabletStats> tablet_stats;
+
+ for (auto& [tmp_rowset_key, tmp_rowset_pb] : tmp_rowsets_meta) {
+ std::string tmp_rowst_data;
+ err = txn->get(tmp_rowset_key, &tmp_rowst_data);
+ if (TxnErrorCode::TXN_KEY_NOT_FOUND == err) {
+ // the tmp rowset has been converted
+ VLOG_DEBUG << "tmp rowset has been converted, key=" << hex(tmp_rowset_key);
+ continue;
+ }
+
+ if (TxnErrorCode::TXN_OK != err) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tmp_rowset_key, txn_id=" << txn_id
+ << " key=" << hex(tmp_rowset_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ if (!tablet_ids.contains(tmp_rowset_pb.tablet_id())) {
+ std::string tablet_idx_key =
+ meta_tablet_idx_key({instance_id, tmp_rowset_pb.tablet_id()});
+ std::string tablet_idx_val;
+ err = txn->get(tablet_idx_key, &tablet_idx_val, true);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get tablet idx, txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key) << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TabletIndexPB tablet_idx_pb;
+ if (!tablet_idx_pb.ParseFromString(tablet_idx_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse tablet idx pb txn_id=" << txn_id
+ << " key=" << hex(tablet_idx_key);
+ msg = ss.str();
+ return;
+ }
+ tablet_ids.emplace(tmp_rowset_pb.tablet_id(), tablet_idx_pb);
+ }
+ const TabletIndexPB& tablet_idx_pb = tablet_ids[tmp_rowset_pb.tablet_id()];
+
+ if (!partition_versions.contains(tmp_rowset_pb.partition_id())) {
+ std::string ver_val;
+ std::string ver_key = partition_version_key(
+ {instance_id, db_id, tablet_idx_pb.table_id(), tmp_rowset_pb.partition_id()});
+ err = txn->get(ver_key, &ver_val);
+ if (TxnErrorCode::TXN_OK != err) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get partiton version, txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse version pb txn_id=" << txn_id << " key=" << hex(ver_key);
+ msg = ss.str();
+ return;
+ }
+ LOG(INFO) << "txn_id=" << txn_id << " key=" << hex(ver_key)
+ << " version_pb:" << version_pb.ShortDebugString();
+ partition_versions.emplace(tmp_rowset_pb.partition_id(), version_pb);
+ }
+
+ const VersionPB& version_pb = partition_versions[tmp_rowset_pb.partition_id()];
+ DCHECK(version_pb.pending_txn_ids_size() > 0);
+ DCHECK_EQ(version_pb.pending_txn_ids(0), txn_id);
+
+ int64_t version = version_pb.has_version() ? (version_pb.version() + 1) : 2;
+
+ std::string rowset_key = meta_rowset_key({instance_id, tmp_rowset_pb.tablet_id(), version});
+ std::string rowset_val;
+ err = txn->get(rowset_key, &rowset_val);
+ if (TxnErrorCode::TXN_OK == err) {
+ // tmp rowset key has been converted
+ continue;
+ }
+
+ if (err != TxnErrorCode::TXN_KEY_NOT_FOUND) {
+ code = cast_as<ErrCategory::READ>(err);
+ ss << "failed to get rowset_key, txn_id=" << txn_id << " key=" << hex(rowset_key)
+ << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+ DCHECK(err == TxnErrorCode::TXN_KEY_NOT_FOUND);
+
+ tmp_rowset_pb.set_start_version(version);
+ tmp_rowset_pb.set_end_version(version);
+
+ rowset_val.clear();
+ if (!tmp_rowset_pb.SerializeToString(&rowset_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize rowset_meta, txn_id=" << txn_id
+ << " key=" << hex(rowset_key);
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(rowset_key, rowset_val);
+ LOG(INFO) << "put rowset_key=" << hex(rowset_key) << " txn_id=" << txn_id
+ << " rowset_size=" << rowset_key.size() + rowset_val.size();
+
+ // Accumulate affected rows
+ auto& stats = tablet_stats[tmp_rowset_pb.tablet_id()];
+ stats.data_size += tmp_rowset_pb.data_disk_size();
+ stats.num_rows += tmp_rowset_pb.num_rows();
+ ++stats.num_rowsets;
+ stats.num_segs += tmp_rowset_pb.num_segments();
+ }
+
+ DCHECK(partition_versions.size() == 1);
+
+ for (auto& [tablet_id, stats] : tablet_stats) {
+ DCHECK(tablet_ids.count(tablet_id));
+ auto& tablet_idx = tablet_ids[tablet_id];
+ StatsTabletKeyInfo info {instance_id, tablet_idx.table_id(), tablet_idx.index_id(),
+ tablet_idx.partition_id(), tablet_id};
+ update_tablet_stats(info, stats, txn, code, msg);
+ if (code != MetaServiceCode::OK) return;
+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg) {
+ // 1. visible txn info
+ // 2. remove running key and put recycle txn key
+
+ std::stringstream ss;
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ ss << "failed to create txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ std::string info_val;
+ const std::string info_key = txn_info_key({instance_id, db_id, txn_id});
+ err = txn->get(info_key, &info_val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = err == TxnErrorCode::TXN_KEY_NOT_FOUND ? MetaServiceCode::TXN_ID_NOT_FOUND
+ : cast_as<ErrCategory::READ>(err);
+ ss << "failed to get txn_info, db_id=" << db_id << " txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ TxnInfoPB txn_info;
+ if (!txn_info.ParseFromString(info_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ ss << "failed to parse txn_info, txn_id=" << txn_id;
+ msg = ss.str();
+ LOG(WARNING) << msg;
+ return;
+ }
+
+ VLOG_DEBUG << "txn_info:" << txn_info.ShortDebugString();
+ DCHECK((txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) ||
+ (txn_info.status() == TxnStatusPB::TXN_STATUS_VISIBLE));
+
+ if (txn_info.status() == TxnStatusPB::TXN_STATUS_COMMITTED) {
+ txn_info.set_status(TxnStatusPB::TXN_STATUS_VISIBLE);
+
+ if (!txn_info.SerializeToString(&info_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize txn_info when saving, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+ txn->put(info_key, info_val);
+ LOG(INFO) << "put info_key=" << hex(info_key) << " txn_id=" << txn_id;
+
+ const std::string running_key = txn_running_key({instance_id, db_id, txn_id});
+ LOG(INFO) << "remove running_key=" << hex(running_key) << " txn_id=" << txn_id;
+ txn->remove(running_key);
+
+ std::string recycle_val;
+ std::string recycle_key = recycle_txn_key({instance_id, db_id, txn_id});
+ RecycleTxnPB recycle_pb;
+ auto now_time = system_clock::now();
+ uint64_t visible_time = duration_cast<milliseconds>(now_time.time_since_epoch()).count();
+ recycle_pb.set_creation_time(visible_time);
+ recycle_pb.set_label(txn_info.label());
+
+ if (!recycle_pb.SerializeToString(&recycle_val)) {
+ code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+ ss << "failed to serialize recycle_pb, txn_id=" << txn_id;
+ msg = ss.str();
+ return;
+ }
+
+ txn->put(recycle_key, recycle_val);
+ LOG(INFO) << "put recycle_key=" << hex(recycle_key) << " txn_id=" << txn_id;
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ ss << "failed to commit kv txn, txn_id=" << txn_id << " err=" << err;
+ msg = ss.str();
+ return;
+ }
+ }
+}
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:296:** 139 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2312363624
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2260209528
<details>
<summary>TPC-H: <b>Total hot run time: 41823 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae48134efcfe7887d2b0b6f2d645e9c4fac57e59, data reload: false
------ Round 1 ----------------------------------
q1 17635 4159 4115 4115
q2 2025 206 201 201
q3 10550 1295 1294 1294
q4 10213 849 987 849
q5 7625 2981 2997 2981
q6 222 138 142 138
q7 1038 620 605 605
q8 9494 1935 1960 1935
q9 8486 6631 6617 6617
q10 8716 3862 3849 3849
q11 434 256 249 249
q12 419 228 230 228
q13 17757 2924 2915 2915
q14 270 246 246 246
q15 522 484 487 484
q16 531 387 384 384
q17 999 962 908 908
q18 8113 7322 7209 7209
q19 1477 1229 1228 1228
q20 566 338 355 338
q21 5345 4764 4769 4764
q22 358 295 286 286
Total cold run time: 112795 ms
Total hot run time: 41823 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4052 4051 4081 4051
q2 324 231 220 220
q3 3010 2993 3147 2993
q4 2003 2049 1962 1962
q5 5622 5545 5444 5444
q6 217 132 134 132
q7 2150 1797 1878 1797
q8 3356 3388 3384 3384
q9 8717 8714 8799 8714
q10 3946 4057 3967 3967
q11 553 441 454 441
q12 714 591 582 582
q13 15293 3119 3124 3119
q14 301 277 284 277
q15 528 485 503 485
q16 456 412 414 412
q17 1800 1713 1700 1700
q18 8320 7788 7838 7788
q19 1735 1722 1714 1714
q20 2056 1857 1882 1857
q21 5775 5434 5437 5434
q22 526 457 488 457
Total cold run time: 71454 ms
Total hot run time: 56930 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1698539283
##########
cloud/src/meta-service/txn_lazy_committer.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "txn_lazy_committer.h"
+
+#include "common/logging.h"
+#include "common/util.h"
+#include "meta-service/keys.h"
+#include "meta-service/meta_service_helper.h"
+
+namespace doris::cloud {
+
+void scan_tmp_rowset(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t* db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>* tmp_rowsets_meta);
+
+void convert_tmp_rowsets(
+ const std::string& instance_id, int64_t txn_id, std::shared_ptr<TxnKv> txn_kv,
+ MetaServiceCode& code, std::string& msg, int64_t db_id,
+ std::vector<std::pair<std::string, doris::RowsetMetaCloudPB>>& tmp_rowsets_meta,
+ std::unordered_map<int64_t, TabletIndexPB>& tablet_ids);
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv, MetaServiceCode& code,
+ std::string& msg);
+
+TxnLazyCommitTask::TxnLazyCommitTask(const std::string& instance_id, int64_t txn_id,
+ std::shared_ptr<TxnKv> txn_kv,
+ TxnLazyCommitter* txn_lazy_committer)
+ : instance_id_(instance_id),
+ txn_id_(txn_id),
+ txn_kv_(txn_kv),
+ txn_lazy_committer_(txn_lazy_committer) {
+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {
Review Comment:
warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TxnLazyCommitTask::commit() {
^
```
<details>
<summary>Additional context</summary>
**cloud/src/meta-service/txn_lazy_committer.cpp:51:** 118 lines including whitespace and comments (threshold 80)
```cpp
void TxnLazyCommitTask::commit() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2304691488
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2304947046
<details>
<summary>TPC-DS: <b>Total hot run time: 192723 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0b1e4a8f000eb4cf1faa30535e050f844fb38a05, data reload: false
query1 1237 881 870 870
query2 6360 1961 1848 1848
query3 10663 3958 3973 3958
query4 59331 26821 23055 23055
query5 5246 492 493 492
query6 411 168 166 166
query7 5739 295 289 289
query8 281 212 213 212
query9 8729 2515 2499 2499
query10 459 280 255 255
query11 17413 15203 15220 15203
query12 164 107 104 104
query13 1555 379 383 379
query14 10928 7385 7359 7359
query15 225 180 177 177
query16 7212 475 435 435
query17 1120 554 565 554
query18 1998 298 315 298
query19 278 150 148 148
query20 120 110 109 109
query21 205 108 105 105
query22 4636 4390 4516 4390
query23 34188 33484 33255 33255
query24 5909 2866 2846 2846
query25 521 398 401 398
query26 633 158 158 158
query27 1681 276 276 276
query28 3571 2063 2041 2041
query29 669 429 419 419
query30 237 165 159 159
query31 937 771 729 729
query32 72 60 58 58
query33 418 286 292 286
query34 867 466 471 466
query35 868 735 747 735
query36 1086 942 931 931
query37 140 85 81 81
query38 3970 3913 3947 3913
query39 1479 1369 1398 1369
query40 205 119 120 119
query41 48 45 47 45
query42 122 99 99 99
query43 521 477 467 467
query44 1104 742 739 739
query45 202 170 168 168
query46 1087 769 734 734
query47 1876 1848 1845 1845
query48 374 294 301 294
query49 781 447 441 441
query50 802 420 410 410
query51 7147 6968 7019 6968
query52 105 89 88 88
query53 250 181 181 181
query54 570 455 467 455
query55 79 79 81 79
query56 311 272 282 272
query57 1199 1076 1061 1061
query58 222 233 238 233
query59 3026 2927 2883 2883
query60 297 270 286 270
query61 128 122 122 122
query62 738 648 668 648
query63 214 185 179 179
query64 2708 2832 2387 2387
query65 3231 3173 3148 3148
query66 677 323 346 323
query67 15581 15336 15198 15198
query68 4526 551 535 535
query69 573 367 287 287
query70 1166 1114 1062 1062
query71 433 261 271 261
query72 7242 2345 2128 2128
query73 762 316 320 316
query74 9544 8863 8808 8808
query75 3508 2671 2669 2669
query76 2062 1043 968 968
query77 712 321 308 308
query78 10939 9324 9485 9324
query79 3131 546 543 543
query80 897 492 493 492
query81 580 230 228 228
query82 768 139 137 137
query83 317 152 146 146
query84 273 75 73 73
query85 695 303 328 303
query86 435 294 293 293
query87 4336 4261 4253 4253
query88 3991 2280 2284 2280
query89 400 281 276 276
query90 1888 191 188 188
query91 119 98 99 98
query92 60 49 51 49
query93 2246 529 534 529
query94 750 282 287 282
query95 353 262 261 261
query96 586 262 268 262
query97 3185 3085 3036 3036
query98 232 203 201 201
query99 1751 1293 1246 1246
Total cold run time: 313621 ms
Total hot run time: 192723 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2304974454
<details>
<summary>ClickBench: <b>Total hot run time: 30.55 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0b1e4a8f000eb4cf1faa30535e050f844fb38a05, data reload: false
query1 0.04 0.04 0.04
query2 0.09 0.04 0.04
query3 0.23 0.06 0.04
query4 1.68 0.11 0.09
query5 0.52 0.49 0.49
query6 1.13 0.75 0.73
query7 0.02 0.02 0.01
query8 0.06 0.05 0.05
query9 0.55 0.50 0.49
query10 0.54 0.55 0.54
query11 0.16 0.12 0.11
query12 0.16 0.12 0.12
query13 0.63 0.58 0.58
query14 0.78 0.79 0.77
query15 0.86 0.83 0.82
query16 0.38 0.38 0.38
query17 1.04 1.00 0.98
query18 0.22 0.21 0.20
query19 1.84 1.69 1.75
query20 0.02 0.01 0.01
query21 15.41 0.66 0.66
query22 4.05 7.73 1.78
query23 18.21 1.43 1.29
query24 2.05 0.25 0.22
query25 0.16 0.08 0.08
query26 0.28 0.18 0.18
query27 0.07 0.08 0.08
query28 13.26 1.02 1.00
query29 12.66 3.35 3.33
query30 0.24 0.06 0.05
query31 2.86 0.40 0.39
query32 3.26 0.47 0.48
query33 2.95 2.97 2.96
query34 16.91 4.40 4.43
query35 4.40 4.42 4.40
query36 0.66 0.48 0.49
query37 0.19 0.16 0.16
query38 0.17 0.15 0.15
query39 0.04 0.03 0.04
query40 0.15 0.12 0.13
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 109.14 s
Total hot run time: 30.55 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2303676968
<details>
<summary>TPC-DS: <b>Total hot run time: 198312 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 740144c09664bd5fcc4a77551e5f4fdf659aa12d, data reload: false
query1 1312 916 889 889
query2 6570 1974 1945 1945
query3 10663 4138 3925 3925
query4 59449 25018 23244 23244
query5 5641 726 744 726
query6 474 210 212 210
query7 5812 337 330 330
query8 516 436 429 429
query9 8674 2574 2533 2533
query10 541 353 343 343
query11 17797 15211 15342 15211
query12 186 132 131 131
query13 1552 444 441 441
query14 11354 7380 7617 7380
query15 252 198 197 197
query16 7440 535 525 525
query17 1192 622 618 618
query18 2060 357 352 352
query19 321 172 178 172
query20 146 140 142 140
query21 251 149 143 143
query22 4599 4423 4599 4423
query23 34739 34092 34046 34046
query24 6176 3018 2978 2978
query25 582 442 427 427
query26 705 189 186 186
query27 1726 334 321 321
query28 3745 2181 2154 2154
query29 733 458 452 452
query30 238 193 190 190
query31 1074 938 871 871
query32 98 81 84 81
query33 539 361 348 348
query34 908 527 518 518
query35 864 799 759 759
query36 1076 998 972 972
query37 163 107 119 107
query38 4050 3917 3968 3917
query39 1547 1473 1453 1453
query40 241 159 160 159
query41 138 137 139 137
query42 133 114 116 114
query43 549 507 522 507
query44 1167 804 811 804
query45 221 194 197 194
query46 1122 792 792 792
query47 1937 1847 1902 1847
query48 428 349 355 349
query49 942 614 594 594
query50 884 474 471 471
query51 7205 7128 7037 7037
query52 120 107 109 107
query53 297 222 226 222
query54 631 542 513 513
query55 96 92 87 87
query56 345 314 324 314
query57 1229 1129 1108 1108
query58 295 309 308 308
query59 3038 2857 2876 2857
query60 383 333 336 333
query61 153 144 152 144
query62 811 703 695 695
query63 264 227 230 227
query64 3375 1882 1893 1882
query65 3257 3241 3241 3241
query66 963 671 670 670
query67 15563 15111 15407 15111
query68 4239 617 587 587
query69 474 325 332 325
query70 1247 1170 1172 1170
query71 408 326 331 326
query72 6427 2415 2112 2112
query73 803 360 362 360
query74 9271 8924 8851 8851
query75 3484 2748 2777 2748
query76 1535 1060 1019 1019
query77 664 443 438 438
query78 10730 9334 9173 9173
query79 1601 581 558 558
query80 1045 635 626 626
query81 601 262 262 262
query82 421 164 162 162
query83 240 208 212 208
query84 285 101 102 101
query85 906 415 358 358
query86 441 322 306 306
query87 4399 4377 4263 4263
query88 3270 2563 2554 2554
query89 441 326 328 326
query90 1869 234 233 233
query91 153 126 124 124
query92 89 82 77 77
query93 1639 555 562 555
query94 808 333 338 333
query95 390 309 296 296
query96 610 290 291 290
query97 3283 3160 3098 3098
query98 253 225 227 225
query99 1859 1350 1337 1337
Total cold run time: 315437 ms
Total hot run time: 198312 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2304158770
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on code in PR #38243:
URL: https://github.com/apache/doris/pull/38243#discussion_r1722598237
##########
be/src/cloud/config.h:
##########
@@ -94,4 +94,6 @@ DECLARE_mBool(save_load_error_log_to_s3);
// the theads which sync the datas which loaded in other clusters
DECLARE_mInt32(sync_load_for_tablets_thread);
+DECLARE_mBool(txn_lazy_commit);
Review Comment:
Add `cloud_` prefix, and add a description too.
##########
cloud/src/meta-service/meta_service.cpp:
##########
@@ -618,11 +649,13 @@ void MetaServiceImpl::create_tablets(::google::protobuf::RpcController* controll
msg = fmt::format("failed to get vault id, vault name={}", name);
return;
}
+ DCHECK(request->has_db_id());
Review Comment:
What happens if the doris FE is the old version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2300026963
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Add lazy commit mechanism for `commit_txn` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38243:
URL: https://github.com/apache/doris/pull/38243#issuecomment-2298075618
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org