You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "JackDrogon (via GitHub)" <gi...@apache.org> on 2023/03/17 01:42:20 UTC
[GitHub] [doris] JackDrogon opened a new pull request, #17881: WIP: support ccr
JackDrogon opened a new pull request, #17881:
URL: https://github.com/apache/doris/pull/17881
## Problem summary
WIP: support ccr feature. Cleaning up commit information, gradually committing
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1563083673
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1549369481
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] JackDrogon commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "JackDrogon (via GitHub)" <gi...@apache.org>.
JackDrogon commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1562757595
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1566659767
<img width="1477" alt="图片" src="https://github.com/apache/doris/assets/7939630/5ee379fc-e711-44b2-87b9-449c8e160a8c">
maybe we need to adjust the log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1547309903
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #17881:
URL: https://github.com/apache/doris/pull/17881#discussion_r1193801045
##########
be/src/http/action/download_binlog_action.cpp:
##########
@@ -0,0 +1,190 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "http/action/download_binlog_action.h"
+
+#include <fmt/format.h>
+#include <fmt/ranges.h>
+
+#include <cstdint>
+#include <stdexcept>
+#include <string_view>
+#include <vector>
+
+#include "common/config.h"
+#include "common/logging.h"
+#include "http/http_channel.h"
+#include "http/http_request.h"
+#include "http/utils.h"
+#include "io/fs/local_file_system.h"
+#include "olap/storage_engine.h"
+#include "olap/tablet.h"
+#include "olap/tablet_manager.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+namespace {
+const std::string kMethodParameter = "method";
+const std::string kTokenParameter = "token";
+const std::string kTabletIdParameter = "tablet_id";
+const std::string kBinlogVersionParameter = "binlog_version";
+const std::string kRowsetIdParameter = "rowset_id";
+const std::string kSegmentIndexParameter = "segment_index";
+
+// get http param, if no value throw exception
+const auto& get_http_param(HttpRequest* req, const std::string& param_name) {
+ const auto& param = req->param(param_name);
+ if (param.empty()) {
+ auto error_msg = fmt::format("parameter {} not specified in url.", param_name);
+ throw std::runtime_error(error_msg);
+ }
+ return param;
+}
+
+auto get_tablet(const std::string& tablet_id_str) {
+ int64_t tablet_id = std::atoll(tablet_id_str.data());
+
+ TabletSharedPtr tablet = StorageEngine::instance()->tablet_manager()->get_tablet(tablet_id);
+ if (tablet == nullptr) {
+ auto error = fmt::format("tablet is not exist, tablet_id={}", tablet_id);
+ LOG(WARNING) << error;
+ throw std::runtime_error(error);
+ }
+
+ return tablet;
+}
+
+// need binlog_version, tablet_id
+void handle_get_binlog_info(HttpRequest* req) {
+ try {
+ const auto& binlog_version = get_http_param(req, kBinlogVersionParameter);
+ const auto& tablet_id = get_http_param(req, kTabletIdParameter);
+ auto tablet = get_tablet(tablet_id);
+
+ const auto& [rowset_id, num_segments] = tablet->get_binlog_info(binlog_version);
+ auto binlog_info_msg = fmt::format("{}:{}", rowset_id, num_segments);
Review Comment:
We should return a json message?
##########
be/src/http/action/monitor_action.cpp:
##########
@@ -1,60 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#include "http/action/monitor_action.h"
-
-#include <glog/logging.h>
-
-#include <sstream>
-#include <utility>
-
-#include "http/http_channel.h"
-#include "http/http_request.h"
-#include "http/http_status.h"
-#include "http/rest_monitor_iface.h"
-
-namespace doris {
-
-const std::string MODULE_KEY = "module";
-
-MonitorAction::MonitorAction() {}
-
-void MonitorAction::register_module(const std::string& name, RestMonitorIface* module) {
- _module_by_name.insert(std::make_pair(name, module));
-}
-
-void MonitorAction::handle(HttpRequest* req) {
- LOG(INFO) << req->debug_string();
- const std::string& module = req->param(MODULE_KEY);
- if (module.empty()) {
- std::string err_msg = "No module params\n";
- HttpChannel::send_reply(req, HttpStatus::OK, err_msg);
- return;
- }
- if (_module_by_name.find(module) == _module_by_name.end()) {
- std::string err_msg = "Unknown module(";
- err_msg += module + ")\n";
- HttpChannel::send_reply(req, HttpStatus::OK, err_msg);
- return;
- }
- std::stringstream ss;
- _module_by_name[module]->debug(ss);
- std::string str = ss.str();
- HttpChannel::send_reply(req, HttpStatus::OK, str);
-}
-
-} // namespace doris
Review Comment:
The action is useless?
##########
gensrc/thrift/FrontendService.thrift:
##########
@@ -523,6 +523,33 @@ struct TLoadTxnBeginResult {
4: optional i64 db_id
}
+struct Coordinator {
+ 1: required string host
+ 2: required i32 port
+}
Review Comment:
duplicated with TNetworkAddress.
##########
be/src/olap/storage_engine.cpp:
##########
@@ -757,6 +769,137 @@ void StorageEngine::_clean_unused_rowset_metas() {
}
}
+void StorageEngine::_gc_binlogs() {
+ auto data_dirs = get_stores();
+ struct tablet_info {
+ std::string tablet_path;
+ int64_t binlog_ttl_ms;
+ };
+ std::unordered_map<int64_t, tablet_info> tablets_info;
+
+ auto get_tablet_info = [&tablets_info, this](int64_t tablet_id) -> const tablet_info& {
+ if (auto iter = tablets_info.find(tablet_id); iter != tablets_info.end()) {
+ return iter->second;
+ }
+
+ auto tablet = tablet_manager()->get_tablet(tablet_id);
+ if (tablet == nullptr) {
+ LOG(WARNING) << "failed to find tablet " << tablet_id;
+ static tablet_info empty_tablet_info;
+ return empty_tablet_info;
+ }
+
+ auto tablet_path = tablet->tablet_path();
+ auto binlog_ttl_ms = tablet->binlog_ttl_ms();
+ tablets_info.emplace(tablet_id, tablet_info {tablet_path, binlog_ttl_ms});
+ return tablets_info[tablet_id];
+ };
+
+ for (auto data_dir : data_dirs) {
+ std::string prefix_key {kBinlogMetaPrefix};
+ OlapMeta* meta = data_dir->get_meta();
+ DCHECK(meta != nullptr);
+
+ auto now = now_ms();
+ int64_t last_tablet_id = 0;
+ std::vector<std::string> wait_for_deleted_binlog_keys;
+ std::vector<std::string> wait_for_deleted_binlog_files;
+ auto add_to_wait_for_deleted_binlog_keys =
+ [&wait_for_deleted_binlog_keys](std::string_view key) {
+ wait_for_deleted_binlog_keys.emplace_back(key);
+ wait_for_deleted_binlog_keys.push_back(get_binlog_data_key_from_meta_key(key));
+ };
+
+ auto add_to_wait_for_deleted = [&add_to_wait_for_deleted_binlog_keys,
+ &wait_for_deleted_binlog_files](
+ std::string_view key, std::string_view tablet_path,
+ int64_t rowset_id, int64_t num_segments) {
+ add_to_wait_for_deleted_binlog_keys(key);
+ for (int64_t i = 0; i < num_segments; ++i) {
+ auto segment_file = fmt::format("{}_{}.dat", rowset_id, i);
Review Comment:
we should not format file name here and should put format in one place, e.g. BetaRowset::segment_file_path
##########
be/src/service/backend_service.cpp:
##########
@@ -373,4 +380,203 @@ void BackendService::clean_trash() {
void BackendService::check_storage_format(TCheckStorageFormatResult& result) {
StorageEngine::instance()->tablet_manager()->get_all_tablets_storage_format(&result);
}
+
+void BackendService::ingest_binlog(TIngestBinlogResult& result,
+ const TIngestBinlogRequest& request) {
+ int64_t txn_id = request.txn_id;
+ // Step 1: get local tablet
+ auto const& local_tablet_id = request.local_tablet_id;
+ auto local_tablet = StorageEngine::instance()->tablet_manager()->get_tablet(local_tablet_id);
+ if (local_tablet == nullptr) {
+ LOG(WARNING) << "tablet " << local_tablet_id << " not found";
+ result.status.__set_status_code(TStatusCode::RUNTIME_ERROR);
Review Comment:
We shoud return TABLET_NOT_FOUND error code?
##########
be/src/olap/olap_meta.cpp:
##########
@@ -54,28 +56,9 @@ using namespace ErrorCode;
const std::string META_POSTFIX = "/meta";
const size_t PREFIX_LENGTH = 4;
-OlapMeta::OlapMeta(const std::string& root_path) : _root_path(root_path), _db(nullptr) {}
+OlapMeta::OlapMeta(const std::string& root_path) : _root_path(root_path) {}
-OlapMeta::~OlapMeta() {
- if (_db != nullptr) {
- for (auto& handle : _handles) {
- _db->DestroyColumnFamilyHandle(handle);
- handle = nullptr;
- }
- rocksdb::Status s = _db->SyncWAL();
- if (!s.ok()) {
- LOG(WARNING) << "rocksdb sync wal failed: " << s.ToString();
- }
- rocksdb::CancelAllBackgroundWork(_db, true);
Review Comment:
I am not sure if we can remove the code.
##########
be/src/service/backend_service.cpp:
##########
@@ -373,4 +380,203 @@ void BackendService::clean_trash() {
void BackendService::check_storage_format(TCheckStorageFormatResult& result) {
StorageEngine::instance()->tablet_manager()->get_all_tablets_storage_format(&result);
}
+
+void BackendService::ingest_binlog(TIngestBinlogResult& result,
+ const TIngestBinlogRequest& request) {
+ int64_t txn_id = request.txn_id;
+ // Step 1: get local tablet
+ auto const& local_tablet_id = request.local_tablet_id;
+ auto local_tablet = StorageEngine::instance()->tablet_manager()->get_tablet(local_tablet_id);
+ if (local_tablet == nullptr) {
+ LOG(WARNING) << "tablet " << local_tablet_id << " not found";
+ result.status.__set_status_code(TStatusCode::RUNTIME_ERROR);
+ result.status.error_msgs.emplace_back(fmt::format("tablet {} not found", local_tablet_id));
+ return;
+ }
+
+ // Step 2: get binlog info
+ auto binlog_api_url = fmt::format("http://{}:{}/api/_binlog/_download", request.remote_host,
+ request.remote_port);
+ constexpr int max_retry = 3;
+
+ auto get_binlog_info_url =
+ fmt::format("{}?method={}&tablet_id={}&binlog_version={}", binlog_api_url,
+ "get_binlog_info", request.remote_tablet_id, request.binlog_version);
+ std::string binlog_info;
+ auto get_binlog_info_cb = [&get_binlog_info_url, &binlog_info](HttpClient* client) {
+ RETURN_IF_ERROR(client->init(get_binlog_info_url));
+ client->set_timeout_ms(10); // 10ms
+ return client->execute(&binlog_info);
+ };
+ auto status = HttpClient::execute_with_retry(max_retry, 1, get_binlog_info_cb);
+ if (!status.ok()) {
+ LOG(WARNING) << "failed to get binlog info from " << get_binlog_info_url
+ << ", status=" << status.to_string();
+ status.to_thrift(&result.status);
+ return;
+ }
+
+ std::vector<std::string> binlog_info_parts = strings::Split(binlog_info, ":");
+ // TODO(Drogon): check binlog info content is right
+ DCHECK(binlog_info_parts.size() == 2);
+ const std::string& rowset_id = binlog_info_parts[0];
+ int64_t num_segments = std::stoll(binlog_info_parts[1]);
+
+ // Step 3: get rowset meta
+ auto get_rowset_meta_url = fmt::format(
+ "{}?method={}&tablet_id={}&rowset_id={}&binlog_version={}", binlog_api_url,
+ "get_rowset_meta", request.remote_tablet_id, rowset_id, request.binlog_version);
+ std::string rowset_meta_str;
+ auto get_rowset_meta_cb = [&get_rowset_meta_url, &rowset_meta_str](HttpClient* client) {
+ RETURN_IF_ERROR(client->init(get_rowset_meta_url));
+ client->set_timeout_ms(10); // 10ms
+ return client->execute(&rowset_meta_str);
+ };
+ status = HttpClient::execute_with_retry(max_retry, 1, get_rowset_meta_cb);
Review Comment:
We can not do http request in bthread context, because pthread running bthread would be blocked.
##########
be/src/olap/storage_engine.cpp:
##########
@@ -757,6 +769,137 @@ void StorageEngine::_clean_unused_rowset_metas() {
}
}
+void StorageEngine::_gc_binlogs() {
+ auto data_dirs = get_stores();
+ struct tablet_info {
+ std::string tablet_path;
+ int64_t binlog_ttl_ms;
+ };
+ std::unordered_map<int64_t, tablet_info> tablets_info;
+
+ auto get_tablet_info = [&tablets_info, this](int64_t tablet_id) -> const tablet_info& {
+ if (auto iter = tablets_info.find(tablet_id); iter != tablets_info.end()) {
+ return iter->second;
+ }
+
+ auto tablet = tablet_manager()->get_tablet(tablet_id);
+ if (tablet == nullptr) {
+ LOG(WARNING) << "failed to find tablet " << tablet_id;
+ static tablet_info empty_tablet_info;
+ return empty_tablet_info;
+ }
+
+ auto tablet_path = tablet->tablet_path();
+ auto binlog_ttl_ms = tablet->binlog_ttl_ms();
+ tablets_info.emplace(tablet_id, tablet_info {tablet_path, binlog_ttl_ms});
+ return tablets_info[tablet_id];
+ };
+
+ for (auto data_dir : data_dirs) {
+ std::string prefix_key {kBinlogMetaPrefix};
+ OlapMeta* meta = data_dir->get_meta();
+ DCHECK(meta != nullptr);
+
+ auto now = now_ms();
+ int64_t last_tablet_id = 0;
+ std::vector<std::string> wait_for_deleted_binlog_keys;
+ std::vector<std::string> wait_for_deleted_binlog_files;
+ auto add_to_wait_for_deleted_binlog_keys =
+ [&wait_for_deleted_binlog_keys](std::string_view key) {
+ wait_for_deleted_binlog_keys.emplace_back(key);
+ wait_for_deleted_binlog_keys.push_back(get_binlog_data_key_from_meta_key(key));
+ };
+
+ auto add_to_wait_for_deleted = [&add_to_wait_for_deleted_binlog_keys,
+ &wait_for_deleted_binlog_files](
+ std::string_view key, std::string_view tablet_path,
+ int64_t rowset_id, int64_t num_segments) {
+ add_to_wait_for_deleted_binlog_keys(key);
+ for (int64_t i = 0; i < num_segments; ++i) {
+ auto segment_file = fmt::format("{}_{}.dat", rowset_id, i);
+ wait_for_deleted_binlog_files.emplace_back(
+ fmt::format("{}/_binlog/{}", tablet_path, segment_file));
Review Comment:
directory should be defined globally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #17881:
URL: https://github.com/apache/doris/pull/17881#discussion_r1204289786
##########
fe/fe-core/src/main/java/org/apache/doris/persist/PartitionPersistInfo.java:
##########
@@ -27,24 +27,36 @@
import org.apache.doris.common.FeMetaVersion;
import org.apache.doris.common.io.Writable;
import org.apache.doris.common.util.RangeUtils;
+import org.apache.doris.persist.gson.GsonUtils;
import com.google.common.collect.Range;
+import com.google.gson.annotations.SerializedName;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
public class PartitionPersistInfo implements Writable {
+ @SerializedName(value = "dbId")
Review Comment:
If using `@SerializedName`, better modify the read/write method too
##########
fe/fe-core/src/main/java/org/apache/doris/binlog/BinlogManager.java:
##########
@@ -0,0 +1,266 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.binlog;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.thrift.TBinlog;
+import org.apache.doris.thrift.TBinlogType;
+import org.apache.doris.thrift.TStatus;
+import org.apache.doris.thrift.TStatusCode;
+
+import com.google.common.collect.Maps;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TMemoryBuffer;
+import org.apache.thrift.transport.TMemoryInputTransport;
+import org.apache.thrift.transport.TTransportException;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+public class BinlogManager {
+ private static final Logger LOG = LogManager.getLogger(BinlogManager.class);
+ private static final int BUFFER_SIZE = 16 * 1024;
+
+ private ReentrantReadWriteLock lock;
+ private Map<Long, DBBinlog> dbBinlogMap;
+ // Pair(commitSeq, timestamp), used for gc
+ // need UpsertRecord to add timestamps for gc
+ private List<Pair<Long, Long>> timestamps;
+
+ public BinlogManager() {
+ lock = new ReentrantReadWriteLock();
+ dbBinlogMap = Maps.newHashMap();
+ timestamps = new ArrayList<Pair<Long, Long>>();
+ }
+
+ private void addBinlog(TBinlog binlog) {
+ LOG.info("add binlog : {}", binlog);
+
+ long dbId = binlog.getDbId();
+ DBBinlog dbBinlog;
+ lock.writeLock().lock();
+ try {
+ dbBinlog = dbBinlogMap.get(dbId);
+ if (dbBinlog == null) {
+ dbBinlog = new DBBinlog(dbId);
+ dbBinlogMap.put(dbId, dbBinlog);
+ }
+ if (binlog.getTimestamp() > 0) {
+ timestamps.add(Pair.of(binlog.getCommitSeq(), binlog.getTimestamp()));
+ }
+ } finally {
+ lock.writeLock().unlock();
+ }
+
+ dbBinlog.addBinlog(binlog);
+ }
+
+ private void addBinlog(long dbId, List<Long> tableIds, long commitSeq, long timestamp, TBinlogType type, String data) {
+ TBinlog binlog = new TBinlog(commitSeq, timestamp, type, dbId, data);
+ if (tableIds != null && !tableIds.isEmpty()) {
+ binlog.setTableIds(tableIds);
+ }
+ addBinlog(binlog);
+ }
+
+ public void addUpsertRecord(UpsertRecord upsertRecord) {
+ LOG.info("add upsert record. upsertRecord: {}", upsertRecord);
+
+ long dbId = upsertRecord.getDbId();
+ List<Long> tableIds = upsertRecord.getAllReleatedTableIds();
+ long commitSeq = upsertRecord.getCommitSeq();
+ long timestamp = upsertRecord.getTimestamp();
+ TBinlogType type = TBinlogType.UPSERT;
+ String data = upsertRecord.toJson();
+
+ addBinlog(dbId, tableIds, commitSeq, timestamp, type, data);
+ }
+
+ public void addAddPartitionRecord(AddPartitionRecord addPartitionRecord) {
+ LOG.info("add partition record. partitionRecord: {}", addPartitionRecord);
+
+ long dbId = addPartitionRecord.getDbId();
+ List<Long> tableIds = new ArrayList<Long>();
+ tableIds.add(addPartitionRecord.getTableId());
+ long commitSeq = addPartitionRecord.getCommitSeq();
+ long timestamp = -1;
+ TBinlogType type = TBinlogType.ADD_PARTITION;
+ String data = addPartitionRecord.toJson();
+
+ addBinlog(dbId, tableIds, commitSeq, timestamp, type, data);
+ }
+
+ // get binlog by dbId, return first binlog.version > version
+ public Pair<TStatus, TBinlog> getBinlog(long dbId, long tableId, long commitSeq) {
+ LOG.info("get binlog. dbId: {}, tableId: {}, commitSeq: {}", dbId, tableId, commitSeq);
+ TStatus status = new TStatus(TStatusCode.OK);
+ lock.readLock().lock();
+ try {
+ DBBinlog dbBinlog = dbBinlogMap.get(dbId);
+ if (dbBinlog == null) {
+ status.setStatusCode(TStatusCode.BINLOG_NOT_FOUND_DB);
+ LOG.warn("dbBinlog not found. dbId: {}", dbId);
+ return Pair.of(status, null);
+ }
+
+ return dbBinlog.getBinlog(tableId, commitSeq);
+ } finally {
+ lock.readLock().unlock();
+ }
+ }
+
+ // gc binlog, remove all binlog timestamp < minTimestamp
+ // TODO(Drogon): get minCommitSeq from timestamps
+ public void gc(long minTimestamp) {
+ lock.writeLock().lock();
+ long minCommitSeq = -1;
+ try {
+ // user iterator to remove element in timestamps
+ for (Iterator<Pair<Long, Long>> iterator = timestamps.iterator(); iterator.hasNext();) {
+ Pair<Long, Long> pair = iterator.next();
+ long commitSeq = pair.first;
+ long timestamp = pair.second;
+
+ if (timestamp >= minTimestamp) {
+ break;
+ }
+
+ iterator.remove();
+ }
+ } finally {
+ lock.writeLock().unlock();
+ }
+
+ if (minCommitSeq == -1) {
+ return;
+ }
+
+ lock.writeLock().lock();
+ }
+
+ private static void writeTBinlogToStream(DataOutputStream dos, TBinlog binlog) throws TException, IOException {
+ TMemoryBuffer buffer = new TMemoryBuffer(BUFFER_SIZE);
+ TBinaryProtocol protocol = new TBinaryProtocol(buffer);
+ binlog.write(protocol);
+ byte[] data = buffer.getArray();
+ dos.writeInt(data.length);
+ dos.write(data);
+ }
+
+
+ // not thread safety, do this without lock
+ public long write(DataOutputStream dos, long checksum) throws IOException {
+ List<TBinlog> binlogs = new ArrayList<TBinlog>();
+ // Step 1: get all binlogs
+ for (DBBinlog dbBinlog : dbBinlogMap.values()) {
+ dbBinlog.getAllBinlogs(binlogs);
+ }
+ // sort binlogs by commitSeq
+ Collections.sort(binlogs, new Comparator<TBinlog>() {
+ @Override
+ public int compare(TBinlog o1, TBinlog o2) {
+ return Long.compare(o1.getCommitSeq(), o2.getCommitSeq());
+ }
+ });
+
+ // Step 2: write binlogs length
+ dos.writeInt(binlogs.size());
+ LOG.info("write binlogs length: {}", binlogs.size());
+
+ // Step 3: write all binlogs to dos
+ // binlog is a thrift type TBinlog
+ for (TBinlog binlog : binlogs) {
+ try {
+ writeTBinlogToStream(dos, binlog);
Review Comment:
Why using thrift to serde binlog?
##########
gensrc/thrift/FrontendService.thrift:
##########
@@ -523,6 +523,33 @@ struct TLoadTxnBeginResult {
4: optional i64 db_id
}
+struct Coordinator {
+ 1: required string host
+ 2: required i32 port
+}
+
+struct TBeginTxnRequest {
+ 1: optional string cluster
Review Comment:
All field should be optional. Do not use requried
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #17881:
URL: https://github.com/apache/doris/pull/17881#discussion_r1192380780
##########
be/src/olap/task/engine_publish_version_task.cpp:
##########
@@ -17,20 +17,45 @@
#include "olap/task/engine_publish_version_task.h"
-#include <util/defer_op.h>
-
#include <map>
+#include "gen_cpp/AgentService_types.h"
#include "olap/data_dir.h"
+#include "olap/olap_define.h"
#include "olap/rowset/rowset_meta_manager.h"
+#include "olap/storage_engine.h"
#include "olap/tablet_manager.h"
+#include "util/defer_op.h"
namespace doris {
using namespace ErrorCode;
using std::map;
+class TabletPublishTxnTask {
+public:
+ TabletPublishTxnTask(EnginePublishVersionTask* engine_task, TabletSharedPtr tablet,
+ RowsetSharedPtr rowset, int64_t partition_id, int64_t transaction_id,
+ Version version, const TabletInfo& tablet_info,
+ std::atomic<int64_t>* total_task_num);
+ ~TabletPublishTxnTask() {}
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```suggestion
~TabletPublishTxnTask() = default;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17881: WIP: support ccr
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1472990489
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1562785005
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1562781172
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #17881:
URL: https://github.com/apache/doris/pull/17881#discussion_r1206364285
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -1988,4 +1988,9 @@ public class Config extends ConfigBase {
+ "the detailed information of all the replicas of the tablet,"
+ " including the specific reason why they are unqueryable, will be printed out."})
public static boolean show_details_for_unaccessible_tablet = false;
+
+ @ConfField(mutable = false, masterOnly = false, expType = ExperimentalType.EXPERIMENTAL, description = {
+ "是否启用binlog特性",
+ "Whether to enable binlog feature"})
+ public static boolean enable_feature_binlog = false;
Review Comment:
```suggestion
public static boolean enable_binlog = false;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #17881:
URL: https://github.com/apache/doris/pull/17881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1563078246
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1547311992
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1562620859
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1563847466
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.81 seconds
stream load tsv: 449 seconds loaded 74807831229 Bytes, about 158 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 80.3 seconds inserted 10000000 Rows, about 124K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230526055808_clickbench_pr_151056.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #17881:
URL: https://github.com/apache/doris/pull/17881#discussion_r1203265141
##########
be/test/olap/binlog_config_test.cpp:
##########
@@ -0,0 +1,150 @@
+#include "olap/binlog_config.h"
Review Comment:
warning: 'olap/binlog_config.h' file not found [clang-diagnostic-error]
```cpp
#include "olap/binlog_config.h"
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #17881:
URL: https://github.com/apache/doris/pull/17881#discussion_r1177295418
##########
gensrc/thrift/FrontendService.thrift:
##########
@@ -847,6 +913,10 @@ service FrontendService {
TLoadTxnCommitResult loadTxnCommit(1: TLoadTxnCommitRequest request)
TLoadTxnRollbackResult loadTxnRollback(1: TLoadTxnRollbackRequest request)
+ TBeginTxnResult beginTxn(1: TBeginTxnRequest request)
Review Comment:
Should we delete shouldTxnBegin in the future? beginTxn and loadTxnBegin are similare.
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -983,6 +998,95 @@ private TLoadTxnBeginResult loadTxnBeginImpl(TLoadTxnBeginRequest request, Strin
return result;
}
+ @Override
+ public TBeginTxnResult beginTxn(TBeginTxnRequest request) throws TException {
+ String clientAddr = getClientAddrAsString();
+ LOG.debug("receive txn begin request: {}, backend: {}", request, clientAddr);
+
+ TBeginTxnResult result = new TBeginTxnResult();
+ TStatus status = new TStatus(TStatusCode.OK);
+ result.setStatus(status);
+ try {
+ TBeginTxnResult tmpRes = beginTxnImpl(request, clientAddr);
+ result.setTxnId(tmpRes.getTxnId()).setDbId(tmpRes.getDbId());
+ } catch (DuplicatedRequestException e) {
+ // this is a duplicate request, just return previous txn id
+ LOG.warn("duplicate request for stream load. request id: {}, txn: {}", e.getDuplicatedRequestId(),
+ e.getTxnId());
+ result.setTxnId(e.getTxnId());
+ } catch (LabelAlreadyUsedException e) {
+ status.setStatusCode(TStatusCode.LABEL_ALREADY_EXISTS);
+ status.addToErrorMsgs(e.getMessage());
+ result.setJobStatus(e.getJobStatus());
+ } catch (UserException e) {
+ LOG.warn("failed to begin: {}", e.getMessage());
+ status.setStatusCode(TStatusCode.ANALYSIS_ERROR);
+ status.addToErrorMsgs(e.getMessage());
+ } catch (Throwable e) {
+ LOG.warn("catch unknown result.", e);
+ status.setStatusCode(TStatusCode.INTERNAL_ERROR);
+ status.addToErrorMsgs(Strings.nullToEmpty(e.getMessage()));
+ return result;
Review Comment:
We can remove this return stmt as above.
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -983,6 +998,95 @@ private TLoadTxnBeginResult loadTxnBeginImpl(TLoadTxnBeginRequest request, Strin
return result;
}
+ @Override
+ public TBeginTxnResult beginTxn(TBeginTxnRequest request) throws TException {
+ String clientAddr = getClientAddrAsString();
+ LOG.debug("receive txn begin request: {}, backend: {}", request, clientAddr);
+
+ TBeginTxnResult result = new TBeginTxnResult();
+ TStatus status = new TStatus(TStatusCode.OK);
+ result.setStatus(status);
+ try {
+ TBeginTxnResult tmpRes = beginTxnImpl(request, clientAddr);
+ result.setTxnId(tmpRes.getTxnId()).setDbId(tmpRes.getDbId());
+ } catch (DuplicatedRequestException e) {
+ // this is a duplicate request, just return previous txn id
+ LOG.warn("duplicate request for stream load. request id: {}, txn: {}", e.getDuplicatedRequestId(),
+ e.getTxnId());
+ result.setTxnId(e.getTxnId());
+ } catch (LabelAlreadyUsedException e) {
+ status.setStatusCode(TStatusCode.LABEL_ALREADY_EXISTS);
+ status.addToErrorMsgs(e.getMessage());
+ result.setJobStatus(e.getJobStatus());
+ } catch (UserException e) {
+ LOG.warn("failed to begin: {}", e.getMessage());
+ status.setStatusCode(TStatusCode.ANALYSIS_ERROR);
+ status.addToErrorMsgs(e.getMessage());
+ } catch (Throwable e) {
+ LOG.warn("catch unknown result.", e);
+ status.setStatusCode(TStatusCode.INTERNAL_ERROR);
+ status.addToErrorMsgs(Strings.nullToEmpty(e.getMessage()));
+ return result;
+ }
+ return result;
+ }
+
+ private TBeginTxnResult beginTxnImpl(TBeginTxnRequest request, String clientIp) throws UserException {
+ String cluster = request.getCluster();
+ if (Strings.isNullOrEmpty(cluster)) {
+ cluster = SystemInfoService.DEFAULT_CLUSTER;
+ }
+
+ // step 1: check auth
+ if (Strings.isNullOrEmpty(request.getToken())) {
+ checkPasswordAndPrivs(cluster, request.getUser(), request.getPasswd(), request.getDb(), request.getTables(),
+ request.getUserIp(), PrivPredicate.LOAD);
+ }
+
+ // step 2: check label
+ if (Strings.isNullOrEmpty(request.getLabel())) {
+ throw new UserException("empty label in begin request");
+ }
+
+ // step 3: check database
+ Env env = Env.getCurrentEnv();
+ String fullDbName = ClusterNamespace.getFullName(cluster, request.getDb());
+ Database db = env.getInternalCatalog().getDbNullable(fullDbName);
+ if (db == null) {
+ String dbName = fullDbName;
+ if (Strings.isNullOrEmpty(request.getCluster())) {
+ dbName = request.getDb();
+ }
+ throw new UserException("unknown database, database=" + dbName);
+ }
+
+ // step 4: fetch all tableIds
+ // lookup tables && convert into tableIdList
+ List<Long> tableIdList = Lists.newArrayList();
+ for (String tblName : request.getTables()) {
+ String fullTblName = ClusterNamespace.getFullName(cluster, tblName);
+ Table table = db.getTableOrMetaException(fullTblName, TableType.OLAP);
+ if (table == null) {
+ throw new UserException("unknown table, table=" + fullTblName);
+ }
+ tableIdList.add(table.getId());
+ }
+
+ // step 5: get timeout
+ long timeoutSecond = request.isSetTimeout() ? request.getTimeout() : Config.stream_load_default_timeout_second;
+
+ // step 6: begin transaction
+ long txnId = Env.getCurrentGlobalTransactionMgr().beginTransaction(
+ db.getId(), tableIdList, request.getLabel(), request.getRequestId(),
+ new TxnCoordinator(TxnSourceType.BE, clientIp),
+ TransactionState.LoadJobSourceType.BACKEND_STREAMING, -1, timeoutSecond);
+
+ // step 7: return result
+ TBeginTxnResult result = new TBeginTxnResult();
+ result.setTxnId(txnId).setDbId(db.getId());
+ return result;
+ }
Review Comment:
Very similar to loadTxnBegin.
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -983,6 +998,95 @@ private TLoadTxnBeginResult loadTxnBeginImpl(TLoadTxnBeginRequest request, Strin
return result;
}
+ @Override
+ public TBeginTxnResult beginTxn(TBeginTxnRequest request) throws TException {
+ String clientAddr = getClientAddrAsString();
+ LOG.debug("receive txn begin request: {}, backend: {}", request, clientAddr);
Review Comment:
maybe client is not a backend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1562677627
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1563790209
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #17881: [Feature-WIP](CCR): Add ccr doris interface (WIP)
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #17881:
URL: https://github.com/apache/doris/pull/17881#issuecomment-1563789136
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org