You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hust-hhb (via GitHub)" <gi...@apache.org> on 2023/08/16 09:11:16 UTC
[GitHub] [doris] hust-hhb opened a new pull request, #23053: [feature](insert) Support wal for group commit insert
hust-hhb opened a new pull request, #23053:
URL: https://github.com/apache/doris/pull/23053
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
support restoring and recovering wal for group commit insert, the relevant pr is https://github.com/apache/doris/pull/22829
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1731857672
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1322297526
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -1139,6 +1144,10 @@ Status VOlapTableSink::prepare(RuntimeState* state) {
}
// Prepare the exprs to run.
RETURN_IF_ERROR(vectorized::VExpr::prepare(_output_vexpr_ctxs, state, _row_desc));
+ if (_group_commit) {
Review Comment:
It's better to move it to group_commit_tablet_sink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1715799650
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1297893524
##########
be/src/runtime/stream_load/stream_load_executor.cpp:
##########
@@ -64,7 +64,8 @@ bvar::LatencyRecorder g_stream_load_begin_txn_latency("stream_load", "begin_txn"
bvar::LatencyRecorder g_stream_load_precommit_txn_latency("stream_load", "precommit_txn");
bvar::LatencyRecorder g_stream_load_commit_txn_latency("stream_load", "commit_txn");
-Status StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<StreamLoadContext> ctx) {
+Status StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<StreamLoadContext> ctx,
Review Comment:
This method will be removed because we are using insert into select to replace stream load.
### Use stream load with SQL
You can add a `sql` parameter to the `Header` to replace the `column_separator`, `line_delimiter`, `where`, `columns` in the previous parameter, which is convenient to use.
```
curl --location-trusted -u user:passwd
[-H "sql: ${load_sql}"...]
-T data.file
-XPUT http://fe_host:http_port/api/{db}/{table}/_stream_load_with_sql
# -- load_sql
# insert into db.table (col, ...) select stream_col, ... from stream("property1"="value1");
# stream
# (
# "column_separator" = ",",
# "format" = "CSV",
# ...
# )
```
Examples：
```
curl --location-trusted -u root: -T test.csv -H "sql:insert into demo.example_tbl_1(user_id, age, cost) select c1, c4, c7 * 2 from stream("format" = "CSV", "column_separator" = "," ) where age >= 30" http://127.0.0.1:28030/api/demo/example_tbl_1/_stream_load_with_sql
```
##########
be/src/runtime/stream_load/stream_load_executor.cpp:
##########
@@ -64,7 +64,8 @@ bvar::LatencyRecorder g_stream_load_begin_txn_latency("stream_load", "begin_txn"
bvar::LatencyRecorder g_stream_load_precommit_txn_latency("stream_load", "precommit_txn");
bvar::LatencyRecorder g_stream_load_commit_txn_latency("stream_load", "commit_txn");
-Status StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<StreamLoadContext> ctx) {
+Status StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<StreamLoadContext> ctx,
Review Comment:
This method will be removed because we are using insert into select to replace stream load.
### Use stream load with SQL
You can add a `sql` parameter to the `Header` to replace the `column_separator`, `line_delimiter`, `where`, `columns` in the previous parameter, which is convenient to use.
```
curl --location-trusted -u user:passwd
[-H "sql: ${load_sql}"...]
-T data.file
-XPUT http://fe_host:http_port/api/{db}/{table}/_stream_load_with_sql
# -- load_sql
# insert into db.table (col, ...) select stream_col, ... from stream("property1"="value1");
# stream
# (
# "column_separator" = ",",
# "format" = "CSV",
# ...
# )
```
Examples：
```
curl --location-trusted -u root: -T test.csv -H "sql:insert into demo.example_tbl_1(user_id, age, cost) select c1, c4, c7 * 2 from stream("format" = "CSV", "column_separator" = "," ) where age >= 30" http://127.0.0.1:28030/api/demo/example_tbl_1/_stream_load_with_sql
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1732851563
TeamCity be ut coverage result:
Function Coverage: 36.31% (8062/22203)
Line Coverage: 28.67% (64884/226312)
Region Coverage: 27.53% (33676/122322)
Branch Coverage: 24.20% (17218/71158)
Coverage Report: http://coverage.selectdb-in.cc/coverage/59f237c9b05dc9704b0fe95bfb72d6897845559a_59f237c9b05dc9704b0fe95bfb72d6897845559a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1732964276
TeamCity be ut coverage result:
Function Coverage: 36.30% (8065/22216)
Line Coverage: 28.65% (64914/226542)
Region Coverage: 27.51% (33692/122456)
Branch Coverage: 24.18% (17227/71240)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7a31799b6fa1465d7f3f8b70607b0cad393066ac_7a31799b6fa1465d7f3f8b70607b0cad393066ac/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1716955774
TeamCity be ut coverage result:
Function Coverage: 37.42% (8064/21551)
Line Coverage: 29.35% (64814/220864)
Region Coverage: 28.29% (33674/119013)
Branch Coverage: 24.74% (17232/69650)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6ef30fe2d8daeef7099d0b6834da5078f88cb3ee_6ef30fe2d8daeef7099d0b6834da5078f88cb3ee/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1334696944
##########
be/src/olap/wal_manager.cpp:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_manager.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include <chrono>
+#include <filesystem>
+
+#include "io/fs/local_file_system.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+WalManager::WalManager(ExecEnv* exec_env, const std::string& wal_dir_list)
+ : _exec_env(exec_env), _stop_background_threads_latch(1) {
+ doris::vectorized::WalReader::string_split(wal_dir_list, ",", _wal_dirs);
+}
+
+WalManager::~WalManager() {
+ _stop_background_threads_latch.count_down();
+ if (_replay_thread) {
+ _replay_thread->join();
+ }
+ LOG(INFO) << "WalManager is destoried";
+}
+void WalManager::stop() {
+ _stop = true;
+ LOG(INFO) << "WalManager is stopped";
+}
+
+Status WalManager::init() {
+ bool exists = false;
+ for (auto wal_dir : _wal_dirs) {
+ std::string tmp_dir = wal_dir + "/tmp";
+ LOG(INFO) << "wal_dir:" << wal_dir << ",tmp_dir:" << tmp_dir;
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(wal_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(wal_dir));
+ }
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(tmp_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(tmp_dir));
+ }
+ RETURN_IF_ERROR(scan_wals(wal_dir));
+ }
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { this->replay(); }, &_replay_thread);
+}
+
+Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id,
+ const std::string& label) {
+ std::string base_path =
+ _wal_dirs.size() == 1 ? _wal_dirs[0] : _wal_dirs[rand() % _wal_dirs.size()];
+ std::stringstream ss;
Review Comment:
warning: variable 'ss' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::stringstream ss = 0;
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_manager.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include <chrono>
+#include <filesystem>
+
+#include "io/fs/local_file_system.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+WalManager::WalManager(ExecEnv* exec_env, const std::string& wal_dir_list)
+ : _exec_env(exec_env), _stop_background_threads_latch(1) {
+ doris::vectorized::WalReader::string_split(wal_dir_list, ",", _wal_dirs);
+}
+
+WalManager::~WalManager() {
+ _stop_background_threads_latch.count_down();
+ if (_replay_thread) {
+ _replay_thread->join();
+ }
+ LOG(INFO) << "WalManager is destoried";
+}
+void WalManager::stop() {
+ _stop = true;
+ LOG(INFO) << "WalManager is stopped";
+}
+
+Status WalManager::init() {
+ bool exists = false;
+ for (auto wal_dir : _wal_dirs) {
+ std::string tmp_dir = wal_dir + "/tmp";
+ LOG(INFO) << "wal_dir:" << wal_dir << ",tmp_dir:" << tmp_dir;
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(wal_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(wal_dir));
+ }
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(tmp_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(tmp_dir));
+ }
+ RETURN_IF_ERROR(scan_wals(wal_dir));
+ }
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { this->replay(); }, &_replay_thread);
+}
+
+Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id,
+ const std::string& label) {
+ std::string base_path =
+ _wal_dirs.size() == 1 ? _wal_dirs[0] : _wal_dirs[rand() % _wal_dirs.size()];
+ std::stringstream ss;
+ ss << base_path << "/" << std::to_string(db_id) << "/" << std::to_string(table_id) << "/"
+ << std::to_string(wal_id) << "_" << label;
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ _wal_path_map.emplace(wal_id, ss.str());
+ }
+ return Status::OK();
+}
+
+Status WalManager::get_wal_path(int64_t wal_id, std::string& wal_path) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = _wal_path_map.find(wal_id);
+ if (it != _wal_path_map.end()) {
+ wal_path = _wal_path_map[wal_id];
+ } else {
+ return Status::InternalError("can not find wal_id {} in wal_path_map", wal_id);
+ }
+ return Status::OK();
+}
+
+Status WalManager::create_wal_reader(const std::string& wal_path,
+ std::shared_ptr<WalReader>& wal_reader) {
+ wal_reader = std::make_shared<WalReader>(wal_path);
+ RETURN_IF_ERROR(wal_reader->init());
+ return Status::OK();
+}
+
+Status WalManager::create_wal_writer(int64_t wal_id, std::shared_ptr<WalWriter>& wal_writer) {
+ std::string wal_path;
Review Comment:
warning: variable 'wal_path' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_path = 0;
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_manager.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include <chrono>
+#include <filesystem>
+
+#include "io/fs/local_file_system.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+WalManager::WalManager(ExecEnv* exec_env, const std::string& wal_dir_list)
+ : _exec_env(exec_env), _stop_background_threads_latch(1) {
+ doris::vectorized::WalReader::string_split(wal_dir_list, ",", _wal_dirs);
+}
+
+WalManager::~WalManager() {
+ _stop_background_threads_latch.count_down();
+ if (_replay_thread) {
+ _replay_thread->join();
+ }
+ LOG(INFO) << "WalManager is destoried";
+}
+void WalManager::stop() {
+ _stop = true;
+ LOG(INFO) << "WalManager is stopped";
+}
+
+Status WalManager::init() {
+ bool exists = false;
+ for (auto wal_dir : _wal_dirs) {
+ std::string tmp_dir = wal_dir + "/tmp";
+ LOG(INFO) << "wal_dir:" << wal_dir << ",tmp_dir:" << tmp_dir;
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(wal_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(wal_dir));
+ }
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(tmp_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(tmp_dir));
+ }
+ RETURN_IF_ERROR(scan_wals(wal_dir));
+ }
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { this->replay(); }, &_replay_thread);
+}
+
+Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id,
+ const std::string& label) {
+ std::string base_path =
+ _wal_dirs.size() == 1 ? _wal_dirs[0] : _wal_dirs[rand() % _wal_dirs.size()];
+ std::stringstream ss;
+ ss << base_path << "/" << std::to_string(db_id) << "/" << std::to_string(table_id) << "/"
+ << std::to_string(wal_id) << "_" << label;
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ _wal_path_map.emplace(wal_id, ss.str());
+ }
+ return Status::OK();
+}
+
+Status WalManager::get_wal_path(int64_t wal_id, std::string& wal_path) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = _wal_path_map.find(wal_id);
+ if (it != _wal_path_map.end()) {
+ wal_path = _wal_path_map[wal_id];
+ } else {
+ return Status::InternalError("can not find wal_id {} in wal_path_map", wal_id);
+ }
+ return Status::OK();
+}
+
+Status WalManager::create_wal_reader(const std::string& wal_path,
+ std::shared_ptr<WalReader>& wal_reader) {
+ wal_reader = std::make_shared<WalReader>(wal_path);
+ RETURN_IF_ERROR(wal_reader->init());
+ return Status::OK();
+}
+
+Status WalManager::create_wal_writer(int64_t wal_id, std::shared_ptr<WalWriter>& wal_writer) {
+ std::string wal_path;
+ RETURN_IF_ERROR(get_wal_path(wal_id, wal_path));
+ std::vector<std::string> path_element;
+ doris::vectorized::WalReader::string_split(wal_path, "/", path_element);
+ std::stringstream ss;
Review Comment:
warning: variable 'ss' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::stringstream ss = 0;
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_manager.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include <chrono>
+#include <filesystem>
+
+#include "io/fs/local_file_system.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+WalManager::WalManager(ExecEnv* exec_env, const std::string& wal_dir_list)
+ : _exec_env(exec_env), _stop_background_threads_latch(1) {
+ doris::vectorized::WalReader::string_split(wal_dir_list, ",", _wal_dirs);
+}
+
+WalManager::~WalManager() {
+ _stop_background_threads_latch.count_down();
+ if (_replay_thread) {
+ _replay_thread->join();
+ }
+ LOG(INFO) << "WalManager is destoried";
+}
+void WalManager::stop() {
+ _stop = true;
+ LOG(INFO) << "WalManager is stopped";
+}
+
+Status WalManager::init() {
+ bool exists = false;
+ for (auto wal_dir : _wal_dirs) {
+ std::string tmp_dir = wal_dir + "/tmp";
+ LOG(INFO) << "wal_dir:" << wal_dir << ",tmp_dir:" << tmp_dir;
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(wal_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(wal_dir));
+ }
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(tmp_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(tmp_dir));
+ }
+ RETURN_IF_ERROR(scan_wals(wal_dir));
+ }
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { this->replay(); }, &_replay_thread);
+}
+
+Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id,
+ const std::string& label) {
+ std::string base_path =
+ _wal_dirs.size() == 1 ? _wal_dirs[0] : _wal_dirs[rand() % _wal_dirs.size()];
+ std::stringstream ss;
+ ss << base_path << "/" << std::to_string(db_id) << "/" << std::to_string(table_id) << "/"
+ << std::to_string(wal_id) << "_" << label;
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ _wal_path_map.emplace(wal_id, ss.str());
+ }
+ return Status::OK();
+}
+
+Status WalManager::get_wal_path(int64_t wal_id, std::string& wal_path) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = _wal_path_map.find(wal_id);
+ if (it != _wal_path_map.end()) {
+ wal_path = _wal_path_map[wal_id];
+ } else {
+ return Status::InternalError("can not find wal_id {} in wal_path_map", wal_id);
+ }
+ return Status::OK();
+}
+
+Status WalManager::create_wal_reader(const std::string& wal_path,
+ std::shared_ptr<WalReader>& wal_reader) {
+ wal_reader = std::make_shared<WalReader>(wal_path);
+ RETURN_IF_ERROR(wal_reader->init());
+ return Status::OK();
+}
+
+Status WalManager::create_wal_writer(int64_t wal_id, std::shared_ptr<WalWriter>& wal_writer) {
+ std::string wal_path;
+ RETURN_IF_ERROR(get_wal_path(wal_id, wal_path));
+ std::vector<std::string> path_element;
+ doris::vectorized::WalReader::string_split(wal_path, "/", path_element);
+ std::stringstream ss;
+ for (int i = 0; i < path_element.size() - 1; i++) {
+ ss << path_element[i] << "/";
+ }
+ std::string base_path = ss.str();
Review Comment:
warning: variable 'base_path' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string base_path = 0 = ss.str();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_manager.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include <chrono>
+#include <filesystem>
+
+#include "io/fs/local_file_system.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+WalManager::WalManager(ExecEnv* exec_env, const std::string& wal_dir_list)
+ : _exec_env(exec_env), _stop_background_threads_latch(1) {
+ doris::vectorized::WalReader::string_split(wal_dir_list, ",", _wal_dirs);
+}
+
+WalManager::~WalManager() {
+ _stop_background_threads_latch.count_down();
+ if (_replay_thread) {
+ _replay_thread->join();
+ }
+ LOG(INFO) << "WalManager is destoried";
+}
+void WalManager::stop() {
+ _stop = true;
+ LOG(INFO) << "WalManager is stopped";
+}
+
+Status WalManager::init() {
+ bool exists = false;
+ for (auto wal_dir : _wal_dirs) {
+ std::string tmp_dir = wal_dir + "/tmp";
+ LOG(INFO) << "wal_dir:" << wal_dir << ",tmp_dir:" << tmp_dir;
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(wal_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(wal_dir));
+ }
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(tmp_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(tmp_dir));
+ }
+ RETURN_IF_ERROR(scan_wals(wal_dir));
+ }
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { this->replay(); }, &_replay_thread);
+}
+
+Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id,
+ const std::string& label) {
+ std::string base_path =
+ _wal_dirs.size() == 1 ? _wal_dirs[0] : _wal_dirs[rand() % _wal_dirs.size()];
+ std::stringstream ss;
+ ss << base_path << "/" << std::to_string(db_id) << "/" << std::to_string(table_id) << "/"
+ << std::to_string(wal_id) << "_" << label;
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ _wal_path_map.emplace(wal_id, ss.str());
+ }
+ return Status::OK();
+}
+
+Status WalManager::get_wal_path(int64_t wal_id, std::string& wal_path) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = _wal_path_map.find(wal_id);
+ if (it != _wal_path_map.end()) {
+ wal_path = _wal_path_map[wal_id];
+ } else {
+ return Status::InternalError("can not find wal_id {} in wal_path_map", wal_id);
+ }
+ return Status::OK();
+}
+
+Status WalManager::create_wal_reader(const std::string& wal_path,
+ std::shared_ptr<WalReader>& wal_reader) {
+ wal_reader = std::make_shared<WalReader>(wal_path);
+ RETURN_IF_ERROR(wal_reader->init());
+ return Status::OK();
+}
+
+Status WalManager::create_wal_writer(int64_t wal_id, std::shared_ptr<WalWriter>& wal_writer) {
+ std::string wal_path;
+ RETURN_IF_ERROR(get_wal_path(wal_id, wal_path));
+ std::vector<std::string> path_element;
+ doris::vectorized::WalReader::string_split(wal_path, "/", path_element);
+ std::stringstream ss;
+ for (int i = 0; i < path_element.size() - 1; i++) {
+ ss << path_element[i] << "/";
+ }
+ std::string base_path = ss.str();
+ bool exists = false;
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(base_path, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(base_path));
+ }
+ LOG(INFO) << "create wal " << wal_path;
+ wal_writer = std::make_shared<WalWriter>(wal_path);
+ RETURN_IF_ERROR(wal_writer->init());
+ return Status::OK();
+}
+
+Status WalManager::scan_wals(const std::string& wal_path) {
+ size_t count = 0;
+ bool exists = true;
+ std::vector<io::FileInfo> dbs;
+ Status st = io::global_local_filesystem()->list(wal_path, false, &dbs, &exists);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed list files for dir=" << wal_path << ", st=" << st.to_string();
+ return st;
+ }
+ for (const auto& db_id : dbs) {
+ if (db_id.is_file) {
+ continue;
+ }
+ std::vector<io::FileInfo> tables;
+ auto db_path = wal_path + "/" + db_id.file_name;
+ st = io::global_local_filesystem()->list(db_path, false, &tables, &exists);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed list files for dir=" << db_path << ", st=" << st.to_string();
+ return st;
+ }
+ for (const auto& table_id : tables) {
+ if (table_id.is_file) {
+ continue;
+ }
+ std::vector<io::FileInfo> wals;
+ auto table_path = db_path + "/" + table_id.file_name;
+ st = io::global_local_filesystem()->list(table_path, false, &wals, &exists);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed list files for dir=" << table_path
+ << ", st=" << st.to_string();
+ return st;
+ }
+ if (wals.size() == 0) {
+ continue;
+ }
+ std::vector<std::string> res;
+ for (const auto& wal : wals) {
+ auto wal_file = table_path + "/" + wal.file_name;
+ res.emplace_back(wal_file);
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ int64_t wal_id = std::strtoll(wal.file_name.c_str(), NULL, 10);
+ _wal_path_map.emplace(wal_id, wal_file);
+ }
+ }
+ st = add_recover_wal(db_id.file_name, table_id.file_name, res);
+ count += res.size();
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed add replay wal, db=" << db_id.file_name
+ << ", table=" << table_id.file_name << ", st=" << st.to_string();
+ return st;
+ }
+ }
+ }
+ LOG(INFO) << "Finish list all wals, size:" << count;
+ return Status::OK();
+}
+
+Status WalManager::replay() {
+ do {
+ if (_stop || _exec_env->master_info() == nullptr) {
+ break;
+ }
+ // port == 0 means not received heartbeat yet
+ while (_exec_env->master_info()->network_address.port == 0) {
+ sleep(1);
+ continue;
+ }
Review Comment:
warning: redundant continue statement at the end of loop statement [readability-redundant-control-flow]
```suggestion
}
```
##########
be/src/olap/wal_reader.cpp:
##########
@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_reader.h"
+
+#include "common/status.h"
+#include "io/fs/file_reader.h"
+#include "io/fs/local_file_system.h"
+#include "io/fs/path.h"
+#include "util/crc32c.h"
+#include "wal_writer.h"
+
+namespace doris {
+
+WalReader::WalReader(const std::string& file_name) : _file_name(file_name), _offset(0) {}
+
+WalReader::~WalReader() {}
+
+Status WalReader::init() {
+ RETURN_IF_ERROR(io::global_local_filesystem()->open_file(_file_name, &file_reader));
+ return Status::OK();
+}
+
+Status WalReader::finalize() {
+ return file_reader->close();
+}
+
+Status WalReader::read_block(PBlock& block) {
+ if (_offset >= file_reader->size()) {
+ return Status::EndOfFile("end of wal file");
+ }
+ size_t bytes_read = 0;
+ uint8_t row_len_buf[WalWriter::LENGTH_SIZE];
+ RETURN_IF_ERROR(
+ file_reader->read_at(_offset, {row_len_buf, WalWriter::LENGTH_SIZE}, &bytes_read));
+ _offset += WalWriter::LENGTH_SIZE;
+ size_t block_len;
+ memcpy(&block_len, row_len_buf, WalWriter::LENGTH_SIZE);
+ // read block
+ std::string block_buf;
Review Comment:
warning: variable 'block_buf' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string block_buf = 0;
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_manager.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include <chrono>
+#include <filesystem>
+
+#include "io/fs/local_file_system.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+WalManager::WalManager(ExecEnv* exec_env, const std::string& wal_dir_list)
+ : _exec_env(exec_env), _stop_background_threads_latch(1) {
+ doris::vectorized::WalReader::string_split(wal_dir_list, ",", _wal_dirs);
+}
+
+WalManager::~WalManager() {
+ _stop_background_threads_latch.count_down();
+ if (_replay_thread) {
+ _replay_thread->join();
+ }
+ LOG(INFO) << "WalManager is destoried";
+}
+void WalManager::stop() {
+ _stop = true;
+ LOG(INFO) << "WalManager is stopped";
+}
+
+Status WalManager::init() {
+ bool exists = false;
+ for (auto wal_dir : _wal_dirs) {
+ std::string tmp_dir = wal_dir + "/tmp";
+ LOG(INFO) << "wal_dir:" << wal_dir << ",tmp_dir:" << tmp_dir;
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(wal_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(wal_dir));
+ }
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(tmp_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(tmp_dir));
+ }
+ RETURN_IF_ERROR(scan_wals(wal_dir));
+ }
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { this->replay(); }, &_replay_thread);
+}
+
+Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id,
+ const std::string& label) {
+ std::string base_path =
+ _wal_dirs.size() == 1 ? _wal_dirs[0] : _wal_dirs[rand() % _wal_dirs.size()];
+ std::stringstream ss;
+ ss << base_path << "/" << std::to_string(db_id) << "/" << std::to_string(table_id) << "/"
+ << std::to_string(wal_id) << "_" << label;
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ _wal_path_map.emplace(wal_id, ss.str());
+ }
+ return Status::OK();
+}
+
+Status WalManager::get_wal_path(int64_t wal_id, std::string& wal_path) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = _wal_path_map.find(wal_id);
+ if (it != _wal_path_map.end()) {
+ wal_path = _wal_path_map[wal_id];
+ } else {
+ return Status::InternalError("can not find wal_id {} in wal_path_map", wal_id);
+ }
+ return Status::OK();
+}
+
+Status WalManager::create_wal_reader(const std::string& wal_path,
+ std::shared_ptr<WalReader>& wal_reader) {
+ wal_reader = std::make_shared<WalReader>(wal_path);
+ RETURN_IF_ERROR(wal_reader->init());
+ return Status::OK();
+}
+
+Status WalManager::create_wal_writer(int64_t wal_id, std::shared_ptr<WalWriter>& wal_writer) {
+ std::string wal_path;
+ RETURN_IF_ERROR(get_wal_path(wal_id, wal_path));
+ std::vector<std::string> path_element;
+ doris::vectorized::WalReader::string_split(wal_path, "/", path_element);
+ std::stringstream ss;
+ for (int i = 0; i < path_element.size() - 1; i++) {
+ ss << path_element[i] << "/";
+ }
+ std::string base_path = ss.str();
+ bool exists = false;
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(base_path, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(base_path));
+ }
+ LOG(INFO) << "create wal " << wal_path;
+ wal_writer = std::make_shared<WalWriter>(wal_path);
+ RETURN_IF_ERROR(wal_writer->init());
+ return Status::OK();
+}
+
+Status WalManager::scan_wals(const std::string& wal_path) {
+ size_t count = 0;
+ bool exists = true;
+ std::vector<io::FileInfo> dbs;
+ Status st = io::global_local_filesystem()->list(wal_path, false, &dbs, &exists);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed list files for dir=" << wal_path << ", st=" << st.to_string();
+ return st;
+ }
+ for (const auto& db_id : dbs) {
+ if (db_id.is_file) {
+ continue;
+ }
+ std::vector<io::FileInfo> tables;
+ auto db_path = wal_path + "/" + db_id.file_name;
+ st = io::global_local_filesystem()->list(db_path, false, &tables, &exists);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed list files for dir=" << db_path << ", st=" << st.to_string();
+ return st;
+ }
+ for (const auto& table_id : tables) {
+ if (table_id.is_file) {
+ continue;
+ }
+ std::vector<io::FileInfo> wals;
+ auto table_path = db_path + "/" + table_id.file_name;
+ st = io::global_local_filesystem()->list(table_path, false, &wals, &exists);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed list files for dir=" << table_path
+ << ", st=" << st.to_string();
+ return st;
+ }
+ if (wals.size() == 0) {
+ continue;
+ }
+ std::vector<std::string> res;
+ for (const auto& wal : wals) {
+ auto wal_file = table_path + "/" + wal.file_name;
+ res.emplace_back(wal_file);
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ int64_t wal_id = std::strtoll(wal.file_name.c_str(), NULL, 10);
+ _wal_path_map.emplace(wal_id, wal_file);
+ }
+ }
+ st = add_recover_wal(db_id.file_name, table_id.file_name, res);
+ count += res.size();
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed add replay wal, db=" << db_id.file_name
+ << ", table=" << table_id.file_name << ", st=" << st.to_string();
+ return st;
+ }
+ }
+ }
+ LOG(INFO) << "Finish list all wals, size:" << count;
+ return Status::OK();
+}
+
+Status WalManager::replay() {
+ do {
+ if (_stop || _exec_env->master_info() == nullptr) {
+ break;
+ }
+ // port == 0 means not received heartbeat yet
+ while (_exec_env->master_info()->network_address.port == 0) {
+ sleep(1);
+ continue;
+ }
+ std::vector<std::string> replay_tables;
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_lock);
+ auto it = _table_map.begin();
+ while (it != _table_map.end()) {
+ if (it->second->size() == 0) {
+ it = _table_map.erase(it);
+ } else {
+ replay_tables.push_back(it->first);
+ it++;
+ }
+ }
+ }
+ for (const auto& table_id : replay_tables) {
+ auto st = _table_map[table_id]->replay_wals();
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed add replay wal on table " << table_id;
+ }
+ }
+ } while (!_stop_background_threads_latch.wait_for(
+ std::chrono::seconds(config::group_commit_replay_wal_retry_interval_seconds)));
+ return Status::OK();
+}
+
+Status WalManager::add_recover_wal(const std::string& db_id, const std::string& table_id,
+ std::vector<std::string> wals) {
+ std::lock_guard<std::shared_mutex> wrlock(_lock);
+ std::shared_ptr<WalTable> table_ptr;
+ auto it = _table_map.find(table_id);
+ if (it == _table_map.end()) {
+ table_ptr = std::make_shared<WalTable>(_exec_env, std::stoll(db_id), std::stoll(table_id));
+ _table_map.emplace(table_id, table_ptr);
+ } else {
+ table_ptr = it->second;
+ }
+ table_ptr->add_wals(wals);
+ return Status::OK();
+}
+
+size_t WalManager::get_wal_table_size(const std::string& table_id) {
+ std::shared_lock rdlock(_lock);
+ auto it = _table_map.find(table_id);
+ if (it != _table_map.end()) {
+ return it->second->size();
+ } else {
+ return 0;
+ }
+}
+
+Status WalManager::delete_wal(int64_t wal_id) {
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ std::string wal_path = _wal_path_map[wal_id];
Review Comment:
warning: variable 'wal_path' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_path = 0 = _wal_path_map[wal_id];
```
##########
be/src/olap/wal_manager.h:
##########
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "common/config.h"
+#include "gen_cpp/FrontendService.h"
Review Comment:
warning: 'gen_cpp/FrontendService.h' file not found [clang-diagnostic-error]
```cpp
#include "gen_cpp/FrontendService.h"
^
```
##########
be/src/olap/wal_reader.cpp:
##########
@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_reader.h"
+
+#include "common/status.h"
+#include "io/fs/file_reader.h"
+#include "io/fs/local_file_system.h"
+#include "io/fs/path.h"
+#include "util/crc32c.h"
+#include "wal_writer.h"
+
+namespace doris {
+
+WalReader::WalReader(const std::string& file_name) : _file_name(file_name), _offset(0) {}
+
+WalReader::~WalReader() {}
+
+Status WalReader::init() {
+ RETURN_IF_ERROR(io::global_local_filesystem()->open_file(_file_name, &file_reader));
+ return Status::OK();
+}
+
+Status WalReader::finalize() {
+ return file_reader->close();
+}
+
+Status WalReader::read_block(PBlock& block) {
+ if (_offset >= file_reader->size()) {
+ return Status::EndOfFile("end of wal file");
+ }
+ size_t bytes_read = 0;
+ uint8_t row_len_buf[WalWriter::LENGTH_SIZE];
+ RETURN_IF_ERROR(
+ file_reader->read_at(_offset, {row_len_buf, WalWriter::LENGTH_SIZE}, &bytes_read));
+ _offset += WalWriter::LENGTH_SIZE;
+ size_t block_len;
Review Comment:
warning: variable 'block_len' is not initialized [cppcoreguidelines-init-variables]
```suggestion
size_t block_len = 0;
```
##########
be/src/olap/wal_table.cpp:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_table.h"
+
+#include <event2/bufferevent.h>
+#include <event2/event.h>
+#include <event2/event_struct.h>
+#include <event2/http.h>
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include "evhttp.h"
+#include "http/action/stream_load.h"
+#include "http/ev_http_server.h"
+#include "http/http_common.h"
+#include "http/http_headers.h"
+#include "http/utils.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_manager.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+
+WalTable::WalTable(ExecEnv* exec_env, int64_t db_id, int64_t table_id)
+ : _exec_env(exec_env), _db_id(db_id), _table_id(table_id) {}
+WalTable::~WalTable() {}
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```suggestion
WalTable::~WalTable() = default;
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_manager.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include <chrono>
+#include <filesystem>
+
+#include "io/fs/local_file_system.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+WalManager::WalManager(ExecEnv* exec_env, const std::string& wal_dir_list)
+ : _exec_env(exec_env), _stop_background_threads_latch(1) {
+ doris::vectorized::WalReader::string_split(wal_dir_list, ",", _wal_dirs);
+}
+
+WalManager::~WalManager() {
+ _stop_background_threads_latch.count_down();
+ if (_replay_thread) {
+ _replay_thread->join();
+ }
+ LOG(INFO) << "WalManager is destoried";
+}
+void WalManager::stop() {
+ _stop = true;
+ LOG(INFO) << "WalManager is stopped";
+}
+
+Status WalManager::init() {
+ bool exists = false;
+ for (auto wal_dir : _wal_dirs) {
+ std::string tmp_dir = wal_dir + "/tmp";
+ LOG(INFO) << "wal_dir:" << wal_dir << ",tmp_dir:" << tmp_dir;
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(wal_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(wal_dir));
+ }
+ RETURN_IF_ERROR(io::global_local_filesystem()->exists(tmp_dir, &exists));
+ if (!exists) {
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(tmp_dir));
+ }
+ RETURN_IF_ERROR(scan_wals(wal_dir));
+ }
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { this->replay(); }, &_replay_thread);
+}
+
+Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id,
+ const std::string& label) {
+ std::string base_path =
Review Comment:
warning: variable 'base_path' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string base_path = 0 =
```
##########
be/src/olap/wal_reader.cpp:
##########
@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_reader.h"
+
+#include "common/status.h"
+#include "io/fs/file_reader.h"
+#include "io/fs/local_file_system.h"
+#include "io/fs/path.h"
+#include "util/crc32c.h"
+#include "wal_writer.h"
+
+namespace doris {
+
+WalReader::WalReader(const std::string& file_name) : _file_name(file_name), _offset(0) {}
+
+WalReader::~WalReader() {}
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```suggestion
WalReader::~WalReader() = default;
```
##########
be/src/olap/wal_reader.cpp:
##########
@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_reader.h"
+
+#include "common/status.h"
+#include "io/fs/file_reader.h"
+#include "io/fs/local_file_system.h"
+#include "io/fs/path.h"
+#include "util/crc32c.h"
+#include "wal_writer.h"
+
+namespace doris {
+
+WalReader::WalReader(const std::string& file_name) : _file_name(file_name), _offset(0) {}
+
+WalReader::~WalReader() {}
+
+Status WalReader::init() {
+ RETURN_IF_ERROR(io::global_local_filesystem()->open_file(_file_name, &file_reader));
+ return Status::OK();
+}
+
+Status WalReader::finalize() {
+ return file_reader->close();
+}
+
+Status WalReader::read_block(PBlock& block) {
+ if (_offset >= file_reader->size()) {
+ return Status::EndOfFile("end of wal file");
+ }
+ size_t bytes_read = 0;
+ uint8_t row_len_buf[WalWriter::LENGTH_SIZE];
+ RETURN_IF_ERROR(
+ file_reader->read_at(_offset, {row_len_buf, WalWriter::LENGTH_SIZE}, &bytes_read));
+ _offset += WalWriter::LENGTH_SIZE;
+ size_t block_len;
+ memcpy(&block_len, row_len_buf, WalWriter::LENGTH_SIZE);
+ // read block
+ std::string block_buf;
+ block_buf.resize(block_len);
+ RETURN_IF_ERROR(file_reader->read_at(_offset, {block_buf.c_str(), block_len}, &bytes_read));
+ _offset += block_len;
+ RETURN_IF_ERROR(_deserialize(block, block_buf));
+ // checksum
+ uint8_t checksum_len_buf[WalWriter::CHECKSUM_SIZE];
+ RETURN_IF_ERROR(file_reader->read_at(_offset, {checksum_len_buf, WalWriter::CHECKSUM_SIZE},
+ &bytes_read));
+ _offset += WalWriter::CHECKSUM_SIZE;
+ uint32_t checksum;
Review Comment:
warning: variable 'checksum' is not initialized [cppcoreguidelines-init-variables]
```suggestion
uint32_t checksum = 0;
```
##########
be/src/olap/wal_writer.cpp:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_writer.h"
+
+#include "io/fs/file_writer.h"
+#include "io/fs/local_file_system.h"
+#include "io/fs/path.h"
+#include "olap/storage_engine.h"
+#include "util/crc32c.h"
+
+namespace doris {
+
+WalWriter::WalWriter(const std::string& file_name) : _file_name(file_name) {}
+
+WalWriter::~WalWriter() {}
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```suggestion
WalWriter::~WalWriter() = default;
```
##########
be/src/olap/wal_table.h:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#pragma once
+#include <mutex>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "gen_cpp/FrontendService.h"
+#include "gen_cpp/FrontendService_types.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "runtime/exec_env.h"
+#include "runtime/stream_load/stream_load_context.h"
+namespace doris {
+class WalTable {
+public:
+ WalTable(ExecEnv* exec_env, int64_t db_id, int64_t table_id);
+ ~WalTable();
+ // <retry_num, start_time_ms, is_doing_replay>
+ using replay_wal_info = std::tuple<int64_t, int64_t, bool>;
+ // used when be start and there are wals need to do recovery
+ void add_wals(std::vector<std::string> wals);
+ Status replay_wals();
+ size_t size();
+
+private:
+ std::pair<int64_t, std::string> get_wal_info(const std::string& wal);
+ std::string get_tmp_path(const std::string wal);
+ Status send_request(int64_t wal_id, const std::string& wal, const std::string& label);
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/olap/wal_table.h:40:** previously declared here
```cpp
private:
^
```
</details>
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -308,9 +311,25 @@ Status GroupCommitTable::_finish_group_commit_load(int64_t db_id, int64_t table_
<< ", instance_id=" << print_id(instance_id)
<< ", executor status=" << status.to_string()
<< ", request commit status=" << st.to_string();
+ if (!prepare_failed) {
+ _exec_env->wal_mgr()->add_wal_path(_db_id, table_id, txn_id, label);
+ std::string wal_path;
Review Comment:
warning: variable 'wal_path' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_path = 0;
```
##########
be/src/runtime/runtime_state.h:
##########
@@ -229,6 +229,12 @@ class RuntimeState {
const std::string& db_name() { return _db_name; }
+ void set_wal_id(int64_t wal_id) { _wal_id = wal_id; }
+
+ int64_t wal_id() { return _wal_id; }
Review Comment:
warning: method 'wal_id' can be made const [readability-make-member-function-const]
```suggestion
int64_t wal_id() const { return _wal_id; }
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -185,16 +186,17 @@ Status GroupCommitTable::get_first_block_load_queue(
Status GroupCommitTable::_create_group_commit_load(
int64_t table_id, std::shared_ptr<LoadBlockQueue>& load_block_queue) {
TStreamLoadPutRequest request;
- std::stringstream ss;
- ss << "insert into " << table_id << " select * from group_commit(\"table_id\"=\"" << table_id
- << "\")";
- request.__set_load_sql(ss.str());
UniqueId load_id = UniqueId::gen_uid();
TUniqueId tload_id;
tload_id.__set_hi(load_id.hi);
tload_id.__set_lo(load_id.lo);
+ std::regex reg("-");
+ std::string label = "group_commit_" + std::regex_replace(load_id.to_string(), reg, "_");
Review Comment:
warning: variable 'label' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string label = 0 = "group_commit_" + std::regex_replace(load_id.to_string(), reg, "_");
```
##########
be/src/olap/wal_writer.cpp:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_writer.h"
+
+#include "io/fs/file_writer.h"
+#include "io/fs/local_file_system.h"
+#include "io/fs/path.h"
+#include "olap/storage_engine.h"
+#include "util/crc32c.h"
+
+namespace doris {
+
+WalWriter::WalWriter(const std::string& file_name) : _file_name(file_name) {}
+
+WalWriter::~WalWriter() {}
+
+Status WalWriter::init() {
+ _batch = config::group_commit_sync_wal_batch;
+ RETURN_IF_ERROR(io::global_local_filesystem()->create_file(_file_name, &_file_writer));
+ return Status::OK();
+}
+
+Status WalWriter::finalize() {
+ RETURN_IF_ERROR(_file_writer->close());
+ return Status::OK();
+}
+
+Status WalWriter::append_blocks(const PBlockArray& blocks) {
+ size_t total_size = 0;
+ for (const auto& block : blocks) {
+ total_size += LENGTH_SIZE + block->ByteSizeLong() + CHECKSUM_SIZE;
+ }
+ std::string binary(total_size, '\0');
+ char* row_binary = binary.data();
Review Comment:
warning: variable 'row_binary' is not initialized [cppcoreguidelines-init-variables]
```suggestion
char* row_binary = nullptr = binary.data();
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -308,9 +311,25 @@ Status GroupCommitTable::_finish_group_commit_load(int64_t db_id, int64_t table_
<< ", instance_id=" << print_id(instance_id)
<< ", executor status=" << status.to_string()
<< ", request commit status=" << st.to_string();
+ if (!prepare_failed) {
+ _exec_env->wal_mgr()->add_wal_path(_db_id, table_id, txn_id, label);
+ std::string wal_path;
+ _exec_env->wal_mgr()->get_wal_path(txn_id, wal_path);
+ _exec_env->wal_mgr()->add_recover_wal(std::to_string(db_id), std::to_string(table_id),
+ std::vector<std::string> {wal_path});
+ }
return st;
}
// TODO handle execute and commit error
+ if (!prepare_failed && !result_status.ok()) {
+ _exec_env->wal_mgr()->add_wal_path(_db_id, table_id, txn_id, label);
+ std::string wal_path;
Review Comment:
warning: variable 'wal_path' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_path = 0;
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -185,16 +186,17 @@ Status GroupCommitTable::get_first_block_load_queue(
Status GroupCommitTable::_create_group_commit_load(
int64_t table_id, std::shared_ptr<LoadBlockQueue>& load_block_queue) {
TStreamLoadPutRequest request;
- std::stringstream ss;
- ss << "insert into " << table_id << " select * from group_commit(\"table_id\"=\"" << table_id
- << "\")";
- request.__set_load_sql(ss.str());
UniqueId load_id = UniqueId::gen_uid();
TUniqueId tload_id;
tload_id.__set_hi(load_id.hi);
tload_id.__set_lo(load_id.lo);
+ std::regex reg("-");
+ std::string label = "group_commit_" + std::regex_replace(load_id.to_string(), reg, "_");
+ std::stringstream ss;
Review Comment:
warning: variable 'ss' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::stringstream ss = 0;
```
##########
be/src/vec/exec/format/wal/wal_reader.cpp:
##########
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "wal_reader.h"
+
+#include "common/logging.h"
+#include "olap/wal_manager.h"
+#include "runtime/runtime_state.h"
+namespace doris::vectorized {
+WalReader::WalReader(RuntimeState* state) : _state(state) {
+ _wal_id = state->wal_id();
+}
+WalReader::~WalReader() {
+ if (_wal_reader.get() != nullptr) {
+ _wal_reader->finalize();
+ }
+}
+Status WalReader::init_reader() {
+ RETURN_IF_ERROR(_state->exec_env()->wal_mgr()->get_wal_path(_wal_id, _wal_path));
+ RETURN_IF_ERROR(_state->exec_env()->wal_mgr()->create_wal_reader(_wal_path, _wal_reader));
+ return Status::OK();
+}
+Status WalReader::get_next_block(Block* block, size_t* read_rows, bool* eof) {
+ PBlock pblock;
+ auto st = _wal_reader->read_block(pblock);
+ if (st.is<ErrorCode::END_OF_FILE>()) {
+ LOG(INFO) << "read eof on wal:" << _wal_path;
+ *read_rows = 0;
+ *eof = true;
+ return Status::OK();
+ }
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed to read wal on path = " << _wal_path;
+ return st;
+ }
+ vectorized::Block tmp_block;
+ tmp_block.deserialize(pblock);
+ block->swap(tmp_block);
+ *read_rows = block->rows();
+ VLOG_DEBUG << "read block rows:" << *read_rows;
+ return Status::OK();
+}
+
+void WalReader::string_split(const std::string& str, const std::string& splits,
+ std::vector<std::string>& res) {
+ if (str == "") return;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (str == "") { return;
}
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,137 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
Review Comment:
warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]
```cpp
#include <gtest/gtest.h>
^
```
##########
be/src/vec/exec/format/wal/wal_reader.cpp:
##########
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "wal_reader.h"
+
+#include "common/logging.h"
+#include "olap/wal_manager.h"
+#include "runtime/runtime_state.h"
+namespace doris::vectorized {
+WalReader::WalReader(RuntimeState* state) : _state(state) {
+ _wal_id = state->wal_id();
+}
+WalReader::~WalReader() {
+ if (_wal_reader.get() != nullptr) {
+ _wal_reader->finalize();
+ }
+}
+Status WalReader::init_reader() {
+ RETURN_IF_ERROR(_state->exec_env()->wal_mgr()->get_wal_path(_wal_id, _wal_path));
+ RETURN_IF_ERROR(_state->exec_env()->wal_mgr()->create_wal_reader(_wal_path, _wal_reader));
+ return Status::OK();
+}
+Status WalReader::get_next_block(Block* block, size_t* read_rows, bool* eof) {
+ PBlock pblock;
+ auto st = _wal_reader->read_block(pblock);
+ if (st.is<ErrorCode::END_OF_FILE>()) {
+ LOG(INFO) << "read eof on wal:" << _wal_path;
+ *read_rows = 0;
+ *eof = true;
+ return Status::OK();
+ }
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed to read wal on path = " << _wal_path;
+ return st;
+ }
+ vectorized::Block tmp_block;
+ tmp_block.deserialize(pblock);
+ block->swap(tmp_block);
+ *read_rows = block->rows();
+ VLOG_DEBUG << "read block rows:" << *read_rows;
+ return Status::OK();
+}
+
+void WalReader::string_split(const std::string& str, const std::string& splits,
+ std::vector<std::string>& res) {
+ if (str == "") return;
+ std::string strs = str + splits;
+ size_t pos = strs.find(splits);
+ int step = splits.size();
+ while (pos != strs.npos) {
+ std::string temp = strs.substr(0, pos);
Review Comment:
warning: variable 'temp' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string temp = 0 = strs.substr(0, pos);
```
##########
be/src/vec/exec/format/wal/wal_reader.cpp:
##########
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "wal_reader.h"
+
+#include "common/logging.h"
+#include "olap/wal_manager.h"
+#include "runtime/runtime_state.h"
+namespace doris::vectorized {
+WalReader::WalReader(RuntimeState* state) : _state(state) {
+ _wal_id = state->wal_id();
+}
+WalReader::~WalReader() {
+ if (_wal_reader.get() != nullptr) {
+ _wal_reader->finalize();
+ }
+}
+Status WalReader::init_reader() {
+ RETURN_IF_ERROR(_state->exec_env()->wal_mgr()->get_wal_path(_wal_id, _wal_path));
+ RETURN_IF_ERROR(_state->exec_env()->wal_mgr()->create_wal_reader(_wal_path, _wal_reader));
+ return Status::OK();
+}
+Status WalReader::get_next_block(Block* block, size_t* read_rows, bool* eof) {
+ PBlock pblock;
+ auto st = _wal_reader->read_block(pblock);
+ if (st.is<ErrorCode::END_OF_FILE>()) {
+ LOG(INFO) << "read eof on wal:" << _wal_path;
+ *read_rows = 0;
+ *eof = true;
+ return Status::OK();
+ }
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed to read wal on path = " << _wal_path;
+ return st;
+ }
+ vectorized::Block tmp_block;
+ tmp_block.deserialize(pblock);
+ block->swap(tmp_block);
+ *read_rows = block->rows();
+ VLOG_DEBUG << "read block rows:" << *read_rows;
+ return Status::OK();
+}
+
+void WalReader::string_split(const std::string& str, const std::string& splits,
+ std::vector<std::string>& res) {
+ if (str == "") return;
+ std::string strs = str + splits;
Review Comment:
warning: variable 'strs' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string strs = 0 = str + splits;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1713017505
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.12 seconds
stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162567873 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1715804672
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.12 seconds
stream load tsv: 606 seconds loaded 74807831229 Bytes, about 117 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162213006 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1712912572
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.9 seconds
stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162271810 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1732868094
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.63 seconds
stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162341815 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1734804136
TeamCity be ut coverage result:
Function Coverage: 36.46% (8162/22388)
Line Coverage: 28.57% (65386/228847)
Region Coverage: 27.43% (33879/123506)
Branch Coverage: 24.07% (17287/71814)
Coverage Report: http://coverage.selectdb-in.cc/coverage/05ac530e3405f7aebd966489d468c99b75d705c6_05ac530e3405f7aebd966489d468c99b75d705c6/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1297246703
##########
be/src/olap/wal_manager.cpp:
##########
@@ -0,0 +1,239 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_manager.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include <chrono>
+#include <filesystem>
+
+#include "io/fs/local_file_system.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+WalManager::WalManager(ExecEnv* exec_env, const std::string& wal_dir_list)
+ : _exec_env(exec_env), _stop_background_threads_latch(1) {
+ doris::vectorized::WalReader::string_split(wal_dir_list, ",", wal_dirs);
+}
+
+WalManager::~WalManager() {
+ _stop_background_threads_latch.count_down();
+ if (_replay_thread) {
+ _replay_thread->join();
+ }
+}
+
+Status WalManager::init() {
+ bool exists = false;
+ for (auto wal_dir : wal_dirs) {
+ std::string tmp_dir = wal_dir + "/tmp";
+ LOG(INFO) << "wal_dir:" << wal_dir << ",tmp_dir:" << tmp_dir;
+ io::global_local_filesystem()->exists(wal_dir, &exists);
+ if (!exists) {
+ io::global_local_filesystem()->create_directory(wal_dir);
+ }
+ io::global_local_filesystem()->exists(tmp_dir, &exists);
+ if (!exists) {
+ io::global_local_filesystem()->create_directory(tmp_dir);
+ }
+ RETURN_IF_ERROR(scan_wals(wal_dir));
+ }
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { this->replay(); }, &_replay_thread);
+}
+
+Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id) {
+ std::string base_path = wal_dirs.size() == 1 ? wal_dirs[0] : wal_dirs[rand() % wal_dirs.size()];
+ std::stringstream ss;
+ ss << base_path << "/" << std::to_string(db_id) << "/" << std::to_string(table_id) << "/"
+ << std::to_string(wal_id);
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ wal_path_map.emplace(wal_id, ss.str());
+ }
+ return Status::OK();
+}
+
+Status WalManager::get_wal_path(int64_t wal_id, std::string& wal_path) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = wal_path_map.find(wal_id);
+ if (it != wal_path_map.end()) {
+ wal_path = wal_path_map[wal_id];
+ } else {
+ return Status::InternalError("can not find wal_id {} in wal_path_map", wal_id);
+ }
+ return Status::OK();
+}
+
+Status WalManager::create_wal_reader(const std::string& wal_path,
+ std::shared_ptr<WalReader>& wal_reader) {
+ wal_reader = std::make_shared<WalReader>(wal_path);
+ RETURN_IF_ERROR(wal_reader->init());
+ return Status::OK();
+}
+
+Status WalManager::create_wal_writer(int64_t wal_id, std::shared_ptr<WalWriter>& wal_writer) {
+ std::string wal_path;
+ get_wal_path(wal_id, wal_path);
+ std::vector<std::string> path_element;
+ doris::vectorized::WalReader::string_split(wal_path, "/", path_element);
+ std::stringstream ss;
+ for (int i = 0; i < path_element.size() - 1; i++) {
+ ss << path_element[i] << "/";
+ }
+ std::string base_path = ss.str();
+ bool exists = false;
+ io::global_local_filesystem()->exists(base_path, &exists);
+ if (!exists) {
+ io::global_local_filesystem()->create_directory(base_path);
Review Comment:
same as above.
##########
be/src/olap/wal_manager.h:
##########
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "common/config.h"
+#include "gen_cpp/FrontendService.h"
+#include "gen_cpp/FrontendService_types.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_table.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/thread.h"
+
+namespace doris {
+class WalManager {
+public:
+ WalManager(ExecEnv* exec_env, const std::string& wal_dir);
+ ~WalManager();
+ Status delete_wal(int64_t wal_id);
+ Status init();
+ Status scan_wals(const std::string& wal_path);
+ Status replay();
+ Status create_wal_reader(const std::string& wal_path, std::shared_ptr<WalReader>& wal_reader);
+ Status create_wal_writer(int64_t wal_id, std::shared_ptr<WalWriter>& wal_writer);
+ Status scan();
+ size_t get_wal_table_size(const std::string& table_id);
+ Status add_recover_wal(const std::string& db_id, const std::string& table_id,
+ std::vector<std::string> wals);
+ Status add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id);
+ Status get_wal_path(int64_t wal_id, std::string& wal_path);
+
+private:
+ ExecEnv* _exec_env;
+ std::shared_mutex _lock;
+ scoped_refptr<Thread> _replay_thread;
+ CountDownLatch _stop_background_threads_latch;
+ std::map<std::string, std::shared_ptr<WalTable>> _table_map;
+ std::vector<std::string> wal_dirs;
+ std::shared_mutex _wal_lock;
+ std::unordered_map<int64_t, std::string> wal_path_map;
Review Comment:
_wal_path_map
##########
be/src/olap/wal_manager.h:
##########
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "common/config.h"
+#include "gen_cpp/FrontendService.h"
+#include "gen_cpp/FrontendService_types.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_table.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/thread.h"
+
+namespace doris {
+class WalManager {
+public:
+ WalManager(ExecEnv* exec_env, const std::string& wal_dir);
+ ~WalManager();
+ Status delete_wal(int64_t wal_id);
+ Status init();
+ Status scan_wals(const std::string& wal_path);
+ Status replay();
+ Status create_wal_reader(const std::string& wal_path, std::shared_ptr<WalReader>& wal_reader);
+ Status create_wal_writer(int64_t wal_id, std::shared_ptr<WalWriter>& wal_writer);
+ Status scan();
+ size_t get_wal_table_size(const std::string& table_id);
+ Status add_recover_wal(const std::string& db_id, const std::string& table_id,
+ std::vector<std::string> wals);
+ Status add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id);
+ Status get_wal_path(int64_t wal_id, std::string& wal_path);
+
+private:
+ ExecEnv* _exec_env;
+ std::shared_mutex _lock;
+ scoped_refptr<Thread> _replay_thread;
+ CountDownLatch _stop_background_threads_latch;
+ std::map<std::string, std::shared_ptr<WalTable>> _table_map;
+ std::vector<std::string> wal_dirs;
Review Comment:
_wal_dirs
##########
be/src/olap/wal_manager.cpp:
##########
@@ -0,0 +1,239 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_manager.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include <chrono>
+#include <filesystem>
+
+#include "io/fs/local_file_system.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+WalManager::WalManager(ExecEnv* exec_env, const std::string& wal_dir_list)
+ : _exec_env(exec_env), _stop_background_threads_latch(1) {
+ doris::vectorized::WalReader::string_split(wal_dir_list, ",", wal_dirs);
+}
+
+WalManager::~WalManager() {
+ _stop_background_threads_latch.count_down();
+ if (_replay_thread) {
+ _replay_thread->join();
+ }
+}
+
+Status WalManager::init() {
+ bool exists = false;
+ for (auto wal_dir : wal_dirs) {
+ std::string tmp_dir = wal_dir + "/tmp";
+ LOG(INFO) << "wal_dir:" << wal_dir << ",tmp_dir:" << tmp_dir;
+ io::global_local_filesystem()->exists(wal_dir, &exists);
+ if (!exists) {
+ io::global_local_filesystem()->create_directory(wal_dir);
Review Comment:
handle error
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -1057,19 +1057,31 @@ private TLoadTxnBeginResult loadTxnBeginImpl(TLoadTxnBeginRequest request, Strin
if (Strings.isNullOrEmpty(request.getLabel())) {
throw new UserException("empty label in begin request");
}
+ OlapTable table;
+ Database db;
// check database
Env env = Env.getCurrentEnv();
- String fullDbName = ClusterNamespace.getFullName(cluster, request.getDb());
- Database db = env.getInternalCatalog().getDbNullable(fullDbName);
- if (db == null) {
- String dbName = fullDbName;
- if (Strings.isNullOrEmpty(request.getCluster())) {
- dbName = request.getDb();
+ if (Strings.isNullOrEmpty(request.getDb()) && request.getTableId() != -1) {
Review Comment:
IfSetTableID && getTableID()
```suggestion
if (Strings.isNullOrEmpty(request.getDb()) && request.getTableId() != -1) {
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -0,0 +1,239 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_manager.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include <chrono>
+#include <filesystem>
+
+#include "io/fs/local_file_system.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+WalManager::WalManager(ExecEnv* exec_env, const std::string& wal_dir_list)
+ : _exec_env(exec_env), _stop_background_threads_latch(1) {
+ doris::vectorized::WalReader::string_split(wal_dir_list, ",", wal_dirs);
+}
+
+WalManager::~WalManager() {
+ _stop_background_threads_latch.count_down();
+ if (_replay_thread) {
+ _replay_thread->join();
+ }
+}
+
+Status WalManager::init() {
+ bool exists = false;
+ for (auto wal_dir : wal_dirs) {
+ std::string tmp_dir = wal_dir + "/tmp";
+ LOG(INFO) << "wal_dir:" << wal_dir << ",tmp_dir:" << tmp_dir;
+ io::global_local_filesystem()->exists(wal_dir, &exists);
+ if (!exists) {
+ io::global_local_filesystem()->create_directory(wal_dir);
+ }
+ io::global_local_filesystem()->exists(tmp_dir, &exists);
+ if (!exists) {
+ io::global_local_filesystem()->create_directory(tmp_dir);
+ }
+ RETURN_IF_ERROR(scan_wals(wal_dir));
+ }
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { this->replay(); }, &_replay_thread);
+}
+
+Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id) {
+ std::string base_path = wal_dirs.size() == 1 ? wal_dirs[0] : wal_dirs[rand() % wal_dirs.size()];
+ std::stringstream ss;
+ ss << base_path << "/" << std::to_string(db_id) << "/" << std::to_string(table_id) << "/"
+ << std::to_string(wal_id);
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ wal_path_map.emplace(wal_id, ss.str());
+ }
+ return Status::OK();
+}
+
+Status WalManager::get_wal_path(int64_t wal_id, std::string& wal_path) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = wal_path_map.find(wal_id);
+ if (it != wal_path_map.end()) {
+ wal_path = wal_path_map[wal_id];
+ } else {
+ return Status::InternalError("can not find wal_id {} in wal_path_map", wal_id);
+ }
+ return Status::OK();
+}
+
+Status WalManager::create_wal_reader(const std::string& wal_path,
+ std::shared_ptr<WalReader>& wal_reader) {
+ wal_reader = std::make_shared<WalReader>(wal_path);
+ RETURN_IF_ERROR(wal_reader->init());
+ return Status::OK();
+}
+
+Status WalManager::create_wal_writer(int64_t wal_id, std::shared_ptr<WalWriter>& wal_writer) {
+ std::string wal_path;
+ get_wal_path(wal_id, wal_path);
Review Comment:
handle return value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1712401318
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.13 seconds
stream load tsv: 535 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162016251 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1731963393
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.08 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162239374 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1732832845
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1734764045
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1734918675
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1712999373
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1712893388
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1713114033
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1712026854
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1314395093
##########
be/src/olap/wal_reader.cpp:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_reader.h"
+
+#include "common/status.h"
+#include "io/fs/file_reader.h"
+#include "io/fs/local_file_system.h"
+#include "io/fs/path.h"
+#include "util/crc32c.h"
+#include "wal_writer.h"
+
+namespace doris {
+
+WalReader::WalReader(const std::string& file_name) : _file_name(file_name), _offset(0) {}
+
+WalReader::~WalReader() {}
+
+Status WalReader::init() {
+ RETURN_IF_ERROR(io::global_local_filesystem()->open_file(_file_name, &file_reader));
+ return Status::OK();
+}
+
+Status WalReader::finalize() {
+ RETURN_IF_ERROR(file_reader->close());
+ return Status::OK();
Review Comment:
return file_reader->close()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1716982058
TeamCity be ut coverage result:
Function Coverage: 37.42% (8064/21551)
Line Coverage: 29.35% (64830/220866)
Region Coverage: 28.29% (33674/119014)
Branch Coverage: 24.76% (17242/69650)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6ef30fe2d8daeef7099d0b6834da5078f88cb3ee_6ef30fe2d8daeef7099d0b6834da5078f88cb3ee/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1713161991
TeamCity be ut coverage result:
Function Coverage: 37.42% (8042/21490)
Line Coverage: 29.38% (64726/220307)
Region Coverage: 28.31% (33580/118635)
Branch Coverage: 24.75% (17186/69430)
Coverage Report: http://coverage.selectdb-in.cc/coverage/053fb2aa8fa0add3ae02c195af6936a092bfdabe_053fb2aa8fa0add3ae02c195af6936a092bfdabe/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1716165537
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1716858179
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #23053:
URL: https://github.com/apache/doris/pull/23053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1734763474
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1732938397
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1718620205
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.19 seconds
stream load tsv: 597 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162308851 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1322301518
##########
be/src/http/utils.cpp:
##########
@@ -72,7 +73,11 @@ bool parse_basic_auth(const HttpRequest& req, std::string* user, std::string* pa
bool parse_basic_auth(const HttpRequest& req, AuthInfo* auth) {
auto& token = req.header("token");
- if (token.empty()) {
+ auto& auth_code_uuid = req.header(HTTP_AUTH_CODE_UUID);
Review Comment:
User can also set "http_code_uuid" when stream load?
This may be unsafe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1704571208
replay wal should also consider schema change, once schema change done, the wal write to table may be failed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1295618871
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gtest/gtest.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
Review Comment:
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
```suggestion
WalManagerTest() = default;
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gtest/gtest.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
Review Comment:
warning: prefer using 'override' or (rarely) 'final' instead of 'virtual' [modernize-use-override]
```suggestion
~WalManagerTest() override {}
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
Review Comment:
warning: 'olap/wal_manager.h' file not found [clang-diagnostic-error]
```cpp
#include "olap/wal_manager.h"
^
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gtest/gtest.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = new WalManager(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_wal_manager);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
+ std::string wal_101 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
+ createWal(wal_100);
+ createWal(wal_101);
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_2_id);
+ std::string wal_200 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_200_id;
+ std::string wal_201 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_201_id;
+ createWal(wal_200);
+ createWal(wal_201);
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size(tb_1_id) > 0 ||
+ _env->_wal_manager->get_wal_table_size(tb_2_id) > 0) {
+ sleep(1);
+ continue;
+ }
Review Comment:
warning: redundant continue statement at the end of loop statement [readability-redundant-control-flow]
```suggestion
}
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/logging.h"
+#include "common/object_pool.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gmock/gmock.h"
+#include "google/protobuf/descriptor.h"
+#include "google/protobuf/service.h"
+#include "gtest/gtest.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "service/brpc.h"
+#include "testutil/desc_tbl_builder.h"
+#include "testutil/test_util.h"
+#include "util/proto_util.h"
+#include "vec/columns/columns_number.h"
+#include "vec/core/types.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+#include "vec/runtime/vdata_stream_recvr.h"
+#include "vec/sink/vdata_stream_sender.h"
+
+#ifndef BE_TEST
+#define BE_TEST
+#endif
+
+using ::testing::_;
+using ::testing::Return;
+using ::testing::SetArgPointee;
+using std::string;
+
+namespace doris {
+
+class WalReaderWriterTest : public testing::Test {
+public:
+ // create a mock cgroup folder
+ virtual void SetUp() { io::global_local_filesystem()->create_directory(_s_test_data_path); }
Review Comment:
warning: prefer using 'override' or (rarely) 'final' instead of 'virtual' [modernize-use-override]
```suggestion
void SetUp() override { io::global_local_filesystem()->create_directory(_s_test_data_path); }
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gtest/gtest.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = new WalManager(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_wal_manager);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
+ std::string wal_101 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
+ createWal(wal_100);
+ createWal(wal_101);
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_2_id);
+ std::string wal_200 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_200_id;
+ std::string wal_201 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_201_id;
+ createWal(wal_200);
+ createWal(wal_201);
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size(tb_1_id) > 0 ||
+ _env->_wal_manager->get_wal_table_size(tb_2_id) > 0) {
+ sleep(1);
+ continue;
+ }
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+ ASSERT_TRUE(!std::filesystem::exists(wal_101));
+ ASSERT_TRUE(!std::filesystem::exists(wal_200));
+ ASSERT_TRUE(!std::filesystem::exists(wal_201));
+}
+
+TEST_F(WalManagerTest, not_need_recovery) {
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
+ createWal(wal_100);
+
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size("1") > 0) {
+ sleep(1);
+ continue;
+ }
Review Comment:
warning: redundant continue statement at the end of loop statement [readability-redundant-control-flow]
```suggestion
}
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/logging.h"
+#include "common/object_pool.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gmock/gmock.h"
+#include "google/protobuf/descriptor.h"
+#include "google/protobuf/service.h"
+#include "gtest/gtest.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "service/brpc.h"
+#include "testutil/desc_tbl_builder.h"
+#include "testutil/test_util.h"
+#include "util/proto_util.h"
+#include "vec/columns/columns_number.h"
+#include "vec/core/types.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+#include "vec/runtime/vdata_stream_recvr.h"
+#include "vec/sink/vdata_stream_sender.h"
+
+#ifndef BE_TEST
+#define BE_TEST
+#endif
+
+using ::testing::_;
+using ::testing::Return;
+using ::testing::SetArgPointee;
+using std::string;
+
+namespace doris {
+
+class WalReaderWriterTest : public testing::Test {
+public:
+ // create a mock cgroup folder
+ virtual void SetUp() { io::global_local_filesystem()->create_directory(_s_test_data_path); }
+
+ // delete the mock cgroup folder
+ virtual void TearDown() { io::global_local_filesystem()->delete_directory(_s_test_data_path); }
Review Comment:
warning: prefer using 'override' or (rarely) 'final' instead of 'virtual' [modernize-use-override]
```suggestion
void TearDown() override { io::global_local_filesystem()->delete_directory(_s_test_data_path); }
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gtest/gtest.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```suggestion
virtual ~WalManagerTest() = default;
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gtest/gtest.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = new WalManager(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_wal_manager);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
+ std::string wal_101 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
+ createWal(wal_100);
+ createWal(wal_101);
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_2_id);
+ std::string wal_200 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_200_id;
+ std::string wal_201 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_201_id;
+ createWal(wal_200);
+ createWal(wal_201);
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size(tb_1_id) > 0 ||
+ _env->_wal_manager->get_wal_table_size(tb_2_id) > 0) {
+ sleep(1);
+ continue;
+ }
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+ ASSERT_TRUE(!std::filesystem::exists(wal_101));
+ ASSERT_TRUE(!std::filesystem::exists(wal_200));
+ ASSERT_TRUE(!std::filesystem::exists(wal_201));
+}
+
+TEST_F(WalManagerTest, not_need_recovery) {
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
+ createWal(wal_100);
+
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size("1") > 0) {
+ sleep(1);
+ continue;
+ }
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+}
+
+TEST_F(WalManagerTest, begin_txn_fail) {
+ Status status = Status::InternalError("TestFail");
+ status.to_thrift(&k_stream_load_begin_result.status);
+ k_check_wal_result.__set_need_recovery(true);
+ config::group_commit_replay_wal_retry_num = 3;
+
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
+ createWal(wal_100);
+
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size("1") > 0) {
+ sleep(1);
+ continue;
+ }
Review Comment:
warning: redundant continue statement at the end of loop statement [readability-redundant-control-flow]
```suggestion
}
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gtest/gtest.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = new WalManager(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_wal_manager);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
+ std::string wal_101 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
+ createWal(wal_100);
+ createWal(wal_101);
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_2_id);
+ std::string wal_200 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_200_id;
+ std::string wal_201 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_201_id;
+ createWal(wal_200);
+ createWal(wal_201);
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size(tb_1_id) > 0 ||
+ _env->_wal_manager->get_wal_table_size(tb_2_id) > 0) {
+ sleep(1);
+ continue;
+ }
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+ ASSERT_TRUE(!std::filesystem::exists(wal_101));
+ ASSERT_TRUE(!std::filesystem::exists(wal_200));
+ ASSERT_TRUE(!std::filesystem::exists(wal_201));
+}
+
+TEST_F(WalManagerTest, not_need_recovery) {
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
+ createWal(wal_100);
+
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size("1") > 0) {
+ sleep(1);
+ continue;
+ }
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+}
+
+TEST_F(WalManagerTest, begin_txn_fail) {
+ Status status = Status::InternalError("TestFail");
+ status.to_thrift(&k_stream_load_begin_result.status);
+ k_check_wal_result.__set_need_recovery(true);
+ config::group_commit_replay_wal_retry_num = 3;
+
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
+ createWal(wal_100);
+
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size("1") > 0) {
+ sleep(1);
+ continue;
+ }
+ std::string tmp_file = tmp_dir + "/" + db_id + "_" + tb_id + "_" + wal_id;
+ ASSERT_TRUE(std::filesystem::exists(tmp_file));
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+}
+
+TEST_F(WalManagerTest, execute_plan_fail) {
+ k_stream_load_plan_status = Status::InternalError("TestFail");
+ k_check_wal_result.__set_need_recovery(true);
+ config::group_commit_replay_wal_retry_num = 3;
+
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
+ createWal(wal_100);
+
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size("1") > 0) {
+ sleep(1);
+ continue;
+ }
Review Comment:
warning: redundant continue statement at the end of loop statement [readability-redundant-control-flow]
```suggestion
}
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
Review Comment:
warning: 'agent/be_exec_version_manager.h' file not found [clang-diagnostic-error]
```cpp
#include "agent/be_exec_version_manager.h"
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1716957719
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1718472760
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1322297968
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -1378,6 +1388,10 @@ Status VOlapTableSink::send(RuntimeState* state, vectorized::Block* input_block,
block.get(), filter_col, block->columns()));
}
}
+ if (_group_commit) {
+ RETURN_IF_ERROR(write_wal(_block_convertor.get(), _tablet_finder.get(), block.get(), state,
Review Comment:
write_wal can be called in group_commit_tablet_sink#handle_block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1716198296
TeamCity be ut coverage result:
Function Coverage: 37.42% (8052/21520)
Line Coverage: 29.34% (64754/220708)
Region Coverage: 28.29% (33659/118960)
Branch Coverage: 24.76% (17239/69630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d0aaeca58c6eee094173bc6e3a6dbf76461f9a3d_d0aaeca58c6eee094173bc6e3a6dbf76461f9a3d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1712405918
TeamCity be ut coverage result:
Function Coverage: 37.40% (8020/21446)
Line Coverage: 29.34% (64459/219716)
Region Coverage: 28.27% (33467/118374)
Branch Coverage: 24.75% (17143/69278)
Coverage Report: http://coverage.selectdb-in.cc/coverage/eb6120ce5dc3688e59d55d17082c795134fcdebb_eb6120ce5dc3688e59d55d17082c795134fcdebb/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1734794227
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.46 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162406771 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1733010344
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1734920077
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1715674785
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1715835708
TeamCity be ut coverage result:
Function Coverage: 37.43% (8054/21520)
Line Coverage: 29.35% (64779/220712)
Region Coverage: 28.31% (33673/118961)
Branch Coverage: 24.77% (17247/69630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f9f236aed095f89e5f1d72843353fb9f9a5a8245_f9f236aed095f89e5f1d72843353fb9f9a5a8245/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1314394842
##########
be/src/olap/wal_table.cpp:
##########
@@ -0,0 +1,397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_table.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include "http/action/stream_load.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_manager.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+
+#ifdef BE_TEST
+TCheckWalResult k_check_wal_result;
+TUpdateWalMapResult k_update_wal_map_result;
+#endif
+
+WalTable::WalTable(ExecEnv* exec_env, int64_t db_id, int64_t table_id)
+ : _exec_env(exec_env), _db_id(db_id), _table_id(table_id) {}
+WalTable::~WalTable() {}
+
+#ifdef BE_TEST
+TStreamLoadPutResult wal_stream_load_put_result;
+#endif
+
+Status WalTable::add_wals(std::vector<std::string> wals) {
+ std::lock_guard<std::mutex> lock(_replay_wal_lock);
+ for (const auto& wal : wals) {
+ LOG(INFO) << "add replay wal " << wal;
+ _replay_wal_map.emplace(wal, replay_wal_info {0, UnixMillis(), false});
+ }
+ return Status::OK();
+}
+Status WalTable::replay_wals() {
+ std::vector<std::string> need_replay_wals;
+ {
+ std::lock_guard<std::mutex> lock(_replay_wal_lock);
+ if (_replay_wal_map.empty()) {
+ return Status::OK();
+ }
+ VLOG_DEBUG << "Start replay wals for db=" << _db_id << ", table=" << _table_id
+ << ", wal size=" << _replay_wal_map.size();
+ for (auto& [wal, info] : _replay_wal_map) {
+ auto& [retry_num, start_ts, replaying] = info;
+ if (replaying) {
+ continue;
+ }
+ if (retry_num >= config::group_commit_replay_wal_retry_num) {
+ LOG(WARNING) << "All replay wal failed, db=" << _db_id << ", table=" << _table_id
+ << ", wal=" << wal
+ << ", retry_num=" << config::group_commit_replay_wal_retry_num;
+ std::string rename_path = get_tmp_path(wal);
+ LOG(INFO) << "rename wal from " << wal << " to " << rename_path;
+ std::rename(wal.c_str(), rename_path.c_str());
+ _replay_wal_map.erase(wal);
+ continue;
+ }
+ if (need_replay(info)) {
+ replaying = true;
+ need_replay_wals.push_back(wal);
+ }
+ }
+ std::sort(need_replay_wals.begin(), need_replay_wals.end());
+ }
+ for (const auto& wal : need_replay_wals) {
+ auto st = replay_wal_internal(wal);
+ if (!st.ok()) {
+ std::lock_guard<std::mutex> lock(_replay_wal_lock);
+ auto it = _replay_wal_map.find(wal);
+ if (it != _replay_wal_map.end()) {
+ auto& [retry_num, start_time, replaying] = it->second;
+ replaying = false;
+ }
+ LOG(WARNING) << "failed replay wal, drop this round, db=" << _db_id
+ << ", table=" << _table_id << ", wal=" << wal << ", st=" << st.to_string();
+ break;
+ }
+ VLOG_NOTICE << "replay wal, db=" << _db_id << ", table=" << _table_id << ", label=" << wal
+ << ", st=" << st.to_string();
+ }
+ return Status::OK();
+}
+
+std::string WalTable::get_tmp_path(const std::string wal) {
+ std::vector<std::string> path_element;
+ doris::vectorized::WalReader::string_split(wal, "/", path_element);
+ std::stringstream ss;
+ int index = 0;
+ while (index < path_element.size() - 3) {
+ ss << path_element[index] << "/";
+ index++;
+ }
+ ss << "tmp/";
+ while (index < path_element.size()) {
+ if (index != path_element.size() - 1) {
+ ss << path_element[index] << "_";
+ } else {
+ ss << path_element[index];
+ }
+ index++;
+ }
+ return ss.str();
+}
+
+bool WalTable::need_replay(const doris::WalTable::replay_wal_info& info) {
+#ifndef BE_TEST
+ auto& [retry_num, start_ts, replaying] = info;
+ auto replay_interval =
+ pow(2, retry_num) * config::group_commit_replay_wal_retry_interval_seconds * 1000;
+ return UnixMillis() - start_ts >= replay_interval;
+#else
+ return true;
+#endif
+}
+
+Status WalTable::replay_wal_internal(const std::string& wal) {
+ LOG(INFO) << "Start replay wal for db=" << _db_id << ", table=" << _table_id << ", wal=" << wal;
+ // start a new stream load
+ {
+ std::lock_guard<std::mutex> lock(_replay_wal_lock);
+ auto it = _replay_wal_map.find(wal);
+ if (it != _replay_wal_map.end()) {
+ auto& [retry_num, start_time, replaying] = it->second;
+ ++retry_num;
+ replaying = true;
+ } else {
+ LOG(WARNING) << "can not find wal in stream load replay map. db=" << _db_id
+ << ", table=" << _table_id << ", wal=" << wal;
+ return Status::OK();
+ }
+ }
+ // check whether wal need to do recover or not
+ bool needRecovery = true;
+ int64_t wal_id = get_wal_id(wal);
+ auto st = check_wal(wal_id, needRecovery);
+ if (!st.ok()) {
+ LOG(WARNING) << "fail to check status for wal=" << wal << ", st=" << st.to_string();
+ return st;
+ } else if (!needRecovery) {
+ LOG(INFO) << "wal is already committed, skip recovery, wal=" << wal_id;
+ _exec_env->wal_mgr()->delete_wal(wal_id);
+
+// RETURN_IF_ERROR(io::global_local_filesystem()->delete_file(std::to_string(wal_id)));
+// LOG(INFO) << "delete wal=" << wal;
+ std::lock_guard<std::mutex> lock(_replay_wal_lock);
+ _replay_wal_map.erase(wal);
+ return Status::OK();
+ }
+ // generate and execute a plan fragment
+ std::shared_ptr<StreamLoadContext> ctx = std::make_shared<StreamLoadContext>(_exec_env);
+ TStreamLoadPutRequest request;
+ st = begin_txn(ctx, wal_id);
+ if (!st.ok()) {
+ LOG(WARNING) << "fail to begin txn files for wal=" << wal << ", st=" << st.to_string();
+ if (ctx->need_rollback) {
+ _exec_env->stream_load_executor()->rollback_txn(ctx.get());
+ ctx->need_rollback = false;
+ }
+ return st;
+ } else {
+ LOG(INFO) << "begin txn files for wal=" << wal << ", st=" << st.to_string();
+ }
+ ctx->wal_path = wal;
+ RETURN_IF_ERROR(update_wal_map(ctx));
+ RETURN_IF_ERROR(generate_stream_load_put_request(ctx, request));
+ RETURN_IF_ERROR(execute_plan_fragment(ctx, request));
+ return Status::OK();
+}
+
+int64_t WalTable::get_wal_id(const std::string& wal) {
+ std::vector<std::string> path_element;
+ doris::vectorized::WalReader::string_split(wal, "/", path_element);
+ int64_t wal_id = std::strtoll(path_element[path_element.size() - 1].c_str(), NULL, 10);
+ return wal_id;
+}
+
+Status WalTable::begin_txn(std::shared_ptr<StreamLoadContext>& stream_load_ctx, int64_t wal_id) {
+ stream_load_ctx->use_streaming = true;
+ stream_load_ctx->load_type = TLoadType::type::MANUL_LOAD;
+ stream_load_ctx->load_src_type = TLoadSourceType::type::RAW;
+ stream_load_ctx->auth.user = "root";
+ stream_load_ctx->auth.passwd = "";
Review Comment:
user and passwd are not "root" and "" in user cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1314410302
##########
be/src/olap/wal_table.cpp:
##########
@@ -0,0 +1,397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_table.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+
+#include "http/action/stream_load.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_manager.h"
+#include "runtime/client_cache.h"
+#include "runtime/fragment_mgr.h"
+#include "runtime/plan_fragment_executor.h"
+#include "util/path_util.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/exec/format/wal/wal_reader.h"
+
+namespace doris {
+
+#ifdef BE_TEST
+TCheckWalResult k_check_wal_result;
+TUpdateWalMapResult k_update_wal_map_result;
+#endif
+
+WalTable::WalTable(ExecEnv* exec_env, int64_t db_id, int64_t table_id)
+ : _exec_env(exec_env), _db_id(db_id), _table_id(table_id) {}
+WalTable::~WalTable() {}
+
+#ifdef BE_TEST
+TStreamLoadPutResult wal_stream_load_put_result;
+#endif
+
+Status WalTable::add_wals(std::vector<std::string> wals) {
+ std::lock_guard<std::mutex> lock(_replay_wal_lock);
+ for (const auto& wal : wals) {
+ LOG(INFO) << "add replay wal " << wal;
+ _replay_wal_map.emplace(wal, replay_wal_info {0, UnixMillis(), false});
+ }
+ return Status::OK();
+}
+Status WalTable::replay_wals() {
+ std::vector<std::string> need_replay_wals;
+ {
+ std::lock_guard<std::mutex> lock(_replay_wal_lock);
+ if (_replay_wal_map.empty()) {
+ return Status::OK();
+ }
+ VLOG_DEBUG << "Start replay wals for db=" << _db_id << ", table=" << _table_id
+ << ", wal size=" << _replay_wal_map.size();
+ for (auto& [wal, info] : _replay_wal_map) {
+ auto& [retry_num, start_ts, replaying] = info;
+ if (replaying) {
+ continue;
+ }
+ if (retry_num >= config::group_commit_replay_wal_retry_num) {
+ LOG(WARNING) << "All replay wal failed, db=" << _db_id << ", table=" << _table_id
+ << ", wal=" << wal
+ << ", retry_num=" << config::group_commit_replay_wal_retry_num;
+ std::string rename_path = get_tmp_path(wal);
+ LOG(INFO) << "rename wal from " << wal << " to " << rename_path;
+ std::rename(wal.c_str(), rename_path.c_str());
+ _replay_wal_map.erase(wal);
+ continue;
+ }
+ if (need_replay(info)) {
+ replaying = true;
+ need_replay_wals.push_back(wal);
+ }
+ }
+ std::sort(need_replay_wals.begin(), need_replay_wals.end());
+ }
+ for (const auto& wal : need_replay_wals) {
+ auto st = replay_wal_internal(wal);
+ if (!st.ok()) {
+ std::lock_guard<std::mutex> lock(_replay_wal_lock);
+ auto it = _replay_wal_map.find(wal);
+ if (it != _replay_wal_map.end()) {
+ auto& [retry_num, start_time, replaying] = it->second;
+ replaying = false;
+ }
+ LOG(WARNING) << "failed replay wal, drop this round, db=" << _db_id
+ << ", table=" << _table_id << ", wal=" << wal << ", st=" << st.to_string();
+ break;
+ }
+ VLOG_NOTICE << "replay wal, db=" << _db_id << ", table=" << _table_id << ", label=" << wal
+ << ", st=" << st.to_string();
+ }
+ return Status::OK();
+}
+
+std::string WalTable::get_tmp_path(const std::string wal) {
+ std::vector<std::string> path_element;
+ doris::vectorized::WalReader::string_split(wal, "/", path_element);
+ std::stringstream ss;
+ int index = 0;
+ while (index < path_element.size() - 3) {
+ ss << path_element[index] << "/";
+ index++;
+ }
+ ss << "tmp/";
+ while (index < path_element.size()) {
+ if (index != path_element.size() - 1) {
+ ss << path_element[index] << "_";
+ } else {
+ ss << path_element[index];
+ }
+ index++;
+ }
+ return ss.str();
+}
+
+bool WalTable::need_replay(const doris::WalTable::replay_wal_info& info) {
+#ifndef BE_TEST
+ auto& [retry_num, start_ts, replaying] = info;
+ auto replay_interval =
+ pow(2, retry_num) * config::group_commit_replay_wal_retry_interval_seconds * 1000;
+ return UnixMillis() - start_ts >= replay_interval;
+#else
+ return true;
+#endif
+}
+
+Status WalTable::replay_wal_internal(const std::string& wal) {
+ LOG(INFO) << "Start replay wal for db=" << _db_id << ", table=" << _table_id << ", wal=" << wal;
+ // start a new stream load
+ {
+ std::lock_guard<std::mutex> lock(_replay_wal_lock);
+ auto it = _replay_wal_map.find(wal);
+ if (it != _replay_wal_map.end()) {
+ auto& [retry_num, start_time, replaying] = it->second;
+ ++retry_num;
+ replaying = true;
+ } else {
+ LOG(WARNING) << "can not find wal in stream load replay map. db=" << _db_id
+ << ", table=" << _table_id << ", wal=" << wal;
+ return Status::OK();
+ }
+ }
+ // check whether wal need to do recover or not
+ bool needRecovery = true;
+ int64_t wal_id = get_wal_id(wal);
+ auto st = check_wal(wal_id, needRecovery);
+ if (!st.ok()) {
+ LOG(WARNING) << "fail to check status for wal=" << wal << ", st=" << st.to_string();
+ return st;
+ } else if (!needRecovery) {
+ LOG(INFO) << "wal is already committed, skip recovery, wal=" << wal_id;
+ _exec_env->wal_mgr()->delete_wal(wal_id);
+
+// RETURN_IF_ERROR(io::global_local_filesystem()->delete_file(std::to_string(wal_id)));
+// LOG(INFO) << "delete wal=" << wal;
+ std::lock_guard<std::mutex> lock(_replay_wal_lock);
+ _replay_wal_map.erase(wal);
+ return Status::OK();
+ }
+ // generate and execute a plan fragment
+ std::shared_ptr<StreamLoadContext> ctx = std::make_shared<StreamLoadContext>(_exec_env);
+ TStreamLoadPutRequest request;
+ st = begin_txn(ctx, wal_id);
+ if (!st.ok()) {
+ LOG(WARNING) << "fail to begin txn files for wal=" << wal << ", st=" << st.to_string();
Review Comment:
should be "fail to begin txn for wal" ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1314413372
##########
be/src/olap/wal_table.h:
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#pragma once
+#include <mutex>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "gen_cpp/FrontendService.h"
+#include "gen_cpp/FrontendService_types.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "runtime/exec_env.h"
+#include "runtime/stream_load/stream_load_context.h"
+namespace doris {
+class WalTable {
+public:
+ WalTable(ExecEnv* exec_env, int64_t db_id, int64_t table_id);
+ ~WalTable();
+ // <retry_num, start_time_ms, is_doing_replay>
+ using replay_wal_info = std::tuple<int64_t, int64_t, bool>;
+ // used when be start and there are wals need to do recovery
+ Status add_wals(std::vector<std::string> wals);
+ Status check_wal(int64_t wal_id, bool& needRecovery);
+ Status replay_wals();
+ Status begin_txn(std::shared_ptr<StreamLoadContext>& stream_load_ctx, int64_t wal_id);
+ Status update_wal_map(std::shared_ptr<StreamLoadContext>& stream_load_ctx);
+ Status generate_stream_load_put_request(std::shared_ptr<StreamLoadContext>& stream_load_ctx,
Review Comment:
move some method to private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1319883637
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <gtest/gtest.h>
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+extern std::string k_response;
+extern std::string k_request_line;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = WalManager::create_shared(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_response = "OK";
+ k_request_line = "\"Status\": \"Success\",\n";
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
+ std::string wal_101 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
Review Comment:
warning: variable 'wal_101' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_101 = 0 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <gtest/gtest.h>
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+extern std::string k_response;
+extern std::string k_request_line;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = WalManager::create_shared(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_response = "OK";
+ k_request_line = "\"Status\": \"Success\",\n";
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
Review Comment:
warning: variable 'wal_100' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_100 = 0 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,137 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/object_pool.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gmock/gmock.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "service/brpc.h"
+#include "testutil/test_util.h"
+#include "util/proto_util.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+#include "vec/runtime/vdata_stream_recvr.h"
+
+using ::testing::_;
+using ::testing::Return;
+using ::testing::SetArgPointee;
+using std::string;
+
+namespace doris {
+
+class WalReaderWriterTest : public testing::Test {
+public:
+ // create a mock cgroup folder
+ virtual void SetUp() { io::global_local_filesystem()->create_directory(_s_test_data_path); }
+
+ // delete the mock cgroup folder
+ virtual void TearDown() { io::global_local_filesystem()->delete_directory(_s_test_data_path); }
+
+ static std::string _s_test_data_path;
+};
+
+std::string WalReaderWriterTest::_s_test_data_path = "./log/wal_reader_writer_test";
+size_t block_rows = 1024;
+
+void covert_block_to_pb(
+ const vectorized::Block& block, PBlock* pblock,
+ segment_v2::CompressionTypePB compression_type = segment_v2::CompressionTypePB::SNAPPY) {
+ size_t uncompressed_bytes = 0;
+ size_t compressed_bytes = 0;
+ Status st = block.serialize(BeExecVersionManager::get_newest_version(), pblock,
Review Comment:
warning: variable 'st' is not initialized [cppcoreguidelines-init-variables]
```suggestion
Status st = 0 = block.serialize(BeExecVersionManager::get_newest_version(), pblock,
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <gtest/gtest.h>
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+extern std::string k_response;
+extern std::string k_request_line;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = WalManager::create_shared(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_response = "OK";
+ k_request_line = "\"Status\": \"Success\",\n";
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
+ std::string wal_101 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
+ createWal(wal_100);
+ createWal(wal_101);
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_2_id);
+ std::string wal_200 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_200_id;
+ std::string wal_201 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_201_id;
+ createWal(wal_200);
+ createWal(wal_201);
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size(tb_1_id) > 0 ||
+ _env->_wal_manager->get_wal_table_size(tb_2_id) > 0) {
+ sleep(1);
+ continue;
+ }
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+ ASSERT_TRUE(!std::filesystem::exists(wal_101));
+ ASSERT_TRUE(!std::filesystem::exists(wal_200));
+ ASSERT_TRUE(!std::filesystem::exists(wal_201));
+}
+
+TEST_F(WalManagerTest, not_need_recovery) {
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
Review Comment:
warning: variable 'wal_100' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_100 = 0 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <gtest/gtest.h>
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+extern std::string k_response;
+extern std::string k_request_line;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = WalManager::create_shared(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_response = "OK";
+ k_request_line = "\"Status\": \"Success\",\n";
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
+ std::string wal_101 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
+ createWal(wal_100);
+ createWal(wal_101);
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_2_id);
+ std::string wal_200 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_200_id;
+ std::string wal_201 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_201_id;
Review Comment:
warning: variable 'wal_201' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_201 = 0 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_201_id;
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,137 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/object_pool.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gmock/gmock.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "service/brpc.h"
+#include "testutil/test_util.h"
+#include "util/proto_util.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+#include "vec/runtime/vdata_stream_recvr.h"
+
+using ::testing::_;
+using ::testing::Return;
+using ::testing::SetArgPointee;
+using std::string;
+
+namespace doris {
+
+class WalReaderWriterTest : public testing::Test {
+public:
+ // create a mock cgroup folder
+ virtual void SetUp() { io::global_local_filesystem()->create_directory(_s_test_data_path); }
+
+ // delete the mock cgroup folder
+ virtual void TearDown() { io::global_local_filesystem()->delete_directory(_s_test_data_path); }
+
+ static std::string _s_test_data_path;
+};
+
+std::string WalReaderWriterTest::_s_test_data_path = "./log/wal_reader_writer_test";
+size_t block_rows = 1024;
+
+void covert_block_to_pb(
+ const vectorized::Block& block, PBlock* pblock,
+ segment_v2::CompressionTypePB compression_type = segment_v2::CompressionTypePB::SNAPPY) {
+ size_t uncompressed_bytes = 0;
+ size_t compressed_bytes = 0;
+ Status st = block.serialize(BeExecVersionManager::get_newest_version(), pblock,
+ &uncompressed_bytes, &compressed_bytes, compression_type);
+ EXPECT_TRUE(st.ok());
+ EXPECT_TRUE(uncompressed_bytes >= compressed_bytes);
+ EXPECT_EQ(compressed_bytes, pblock->column_values().size());
+
+ const vectorized::ColumnWithTypeAndName& type_and_name =
+ block.get_columns_with_type_and_name()[0];
+ EXPECT_EQ(type_and_name.name, pblock->column_metas()[0].name());
+}
+
+void generate_block(PBlock& pblock, int row_index) {
+ auto vec = vectorized::ColumnVector<int32_t>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < block_rows; ++i) {
+ data.push_back(i + row_index);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ vectorized::Block block({type_and_name});
+ covert_block_to_pb(block, &pblock, segment_v2::CompressionTypePB::SNAPPY);
+}
+
+TEST_F(WalReaderWriterTest, TestWriteAndRead1) {
+ std::string file_name = _s_test_data_path + "/abcd123.txt";
+ auto wal_writer = WalWriter(file_name);
+ wal_writer.init();
+ size_t file_len = 0;
+ int64_t file_size = -1;
+ // add 1 block
+ {
+ PBlock pblock;
+ generate_block(pblock, 0);
+
+ EXPECT_EQ(Status::OK(), wal_writer.append_blocks(std::vector<PBlock*> {&pblock}));
+ file_len += pblock.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+ io::global_local_filesystem()->file_size(file_name, &file_size);
+ EXPECT_EQ(file_len, file_size);
+ }
+ // add 2 block
+ {
+ PBlock pblock;
+ generate_block(pblock, 1024);
+ file_len += pblock.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+
+ PBlock pblock1;
+ generate_block(pblock1, 2048);
+ file_len += pblock1.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+
+ EXPECT_EQ(Status::OK(), wal_writer.append_blocks(std::vector<PBlock*> {&pblock, &pblock1}));
+ io::global_local_filesystem()->file_size(file_name, &file_size);
+ EXPECT_EQ(file_len, file_size);
+ }
+ wal_writer.finalize();
+ // read block
+ auto wal_reader = WalReader(file_name);
+ wal_reader.init();
+ auto block_count = 0;
+ while (true) {
+ doris::PBlock pblock;
Review Comment:
warning: variable 'pblock' is not initialized [cppcoreguidelines-init-variables]
```suggestion
doris::PBlock pblock = 0;
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <gtest/gtest.h>
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+extern std::string k_response;
+extern std::string k_request_line;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = WalManager::create_shared(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_response = "OK";
+ k_request_line = "\"Status\": \"Success\",\n";
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
+ std::string wal_101 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
+ createWal(wal_100);
+ createWal(wal_101);
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_2_id);
+ std::string wal_200 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_200_id;
Review Comment:
warning: variable 'wal_200' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_200 = 0 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_200_id;
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,137 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/object_pool.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gmock/gmock.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "service/brpc.h"
+#include "testutil/test_util.h"
+#include "util/proto_util.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+#include "vec/runtime/vdata_stream_recvr.h"
+
+using ::testing::_;
+using ::testing::Return;
+using ::testing::SetArgPointee;
+using std::string;
+
+namespace doris {
+
+class WalReaderWriterTest : public testing::Test {
+public:
+ // create a mock cgroup folder
+ virtual void SetUp() { io::global_local_filesystem()->create_directory(_s_test_data_path); }
+
+ // delete the mock cgroup folder
+ virtual void TearDown() { io::global_local_filesystem()->delete_directory(_s_test_data_path); }
+
+ static std::string _s_test_data_path;
+};
+
+std::string WalReaderWriterTest::_s_test_data_path = "./log/wal_reader_writer_test";
+size_t block_rows = 1024;
+
+void covert_block_to_pb(
+ const vectorized::Block& block, PBlock* pblock,
+ segment_v2::CompressionTypePB compression_type = segment_v2::CompressionTypePB::SNAPPY) {
+ size_t uncompressed_bytes = 0;
+ size_t compressed_bytes = 0;
+ Status st = block.serialize(BeExecVersionManager::get_newest_version(), pblock,
+ &uncompressed_bytes, &compressed_bytes, compression_type);
+ EXPECT_TRUE(st.ok());
+ EXPECT_TRUE(uncompressed_bytes >= compressed_bytes);
+ EXPECT_EQ(compressed_bytes, pblock->column_values().size());
+
+ const vectorized::ColumnWithTypeAndName& type_and_name =
+ block.get_columns_with_type_and_name()[0];
+ EXPECT_EQ(type_and_name.name, pblock->column_metas()[0].name());
+}
+
+void generate_block(PBlock& pblock, int row_index) {
+ auto vec = vectorized::ColumnVector<int32_t>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < block_rows; ++i) {
+ data.push_back(i + row_index);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ vectorized::Block block({type_and_name});
+ covert_block_to_pb(block, &pblock, segment_v2::CompressionTypePB::SNAPPY);
+}
+
+TEST_F(WalReaderWriterTest, TestWriteAndRead1) {
+ std::string file_name = _s_test_data_path + "/abcd123.txt";
+ auto wal_writer = WalWriter(file_name);
+ wal_writer.init();
+ size_t file_len = 0;
+ int64_t file_size = -1;
+ // add 1 block
+ {
+ PBlock pblock;
+ generate_block(pblock, 0);
+
+ EXPECT_EQ(Status::OK(), wal_writer.append_blocks(std::vector<PBlock*> {&pblock}));
+ file_len += pblock.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+ io::global_local_filesystem()->file_size(file_name, &file_size);
+ EXPECT_EQ(file_len, file_size);
+ }
+ // add 2 block
+ {
+ PBlock pblock;
+ generate_block(pblock, 1024);
+ file_len += pblock.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+
+ PBlock pblock1;
Review Comment:
warning: variable 'pblock1' is not initialized [cppcoreguidelines-init-variables]
```suggestion
PBlock pblock1 = 0;
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,137 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/object_pool.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gmock/gmock.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "service/brpc.h"
+#include "testutil/test_util.h"
+#include "util/proto_util.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+#include "vec/runtime/vdata_stream_recvr.h"
+
+using ::testing::_;
+using ::testing::Return;
+using ::testing::SetArgPointee;
+using std::string;
+
+namespace doris {
+
+class WalReaderWriterTest : public testing::Test {
+public:
+ // create a mock cgroup folder
+ virtual void SetUp() { io::global_local_filesystem()->create_directory(_s_test_data_path); }
+
+ // delete the mock cgroup folder
+ virtual void TearDown() { io::global_local_filesystem()->delete_directory(_s_test_data_path); }
+
+ static std::string _s_test_data_path;
+};
+
+std::string WalReaderWriterTest::_s_test_data_path = "./log/wal_reader_writer_test";
+size_t block_rows = 1024;
+
+void covert_block_to_pb(
+ const vectorized::Block& block, PBlock* pblock,
+ segment_v2::CompressionTypePB compression_type = segment_v2::CompressionTypePB::SNAPPY) {
+ size_t uncompressed_bytes = 0;
+ size_t compressed_bytes = 0;
+ Status st = block.serialize(BeExecVersionManager::get_newest_version(), pblock,
+ &uncompressed_bytes, &compressed_bytes, compression_type);
+ EXPECT_TRUE(st.ok());
+ EXPECT_TRUE(uncompressed_bytes >= compressed_bytes);
+ EXPECT_EQ(compressed_bytes, pblock->column_values().size());
+
+ const vectorized::ColumnWithTypeAndName& type_and_name =
+ block.get_columns_with_type_and_name()[0];
+ EXPECT_EQ(type_and_name.name, pblock->column_metas()[0].name());
+}
+
+void generate_block(PBlock& pblock, int row_index) {
+ auto vec = vectorized::ColumnVector<int32_t>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < block_rows; ++i) {
+ data.push_back(i + row_index);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ vectorized::Block block({type_and_name});
+ covert_block_to_pb(block, &pblock, segment_v2::CompressionTypePB::SNAPPY);
+}
+
+TEST_F(WalReaderWriterTest, TestWriteAndRead1) {
+ std::string file_name = _s_test_data_path + "/abcd123.txt";
+ auto wal_writer = WalWriter(file_name);
+ wal_writer.init();
+ size_t file_len = 0;
+ int64_t file_size = -1;
+ // add 1 block
+ {
+ PBlock pblock;
+ generate_block(pblock, 0);
+
+ EXPECT_EQ(Status::OK(), wal_writer.append_blocks(std::vector<PBlock*> {&pblock}));
+ file_len += pblock.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+ io::global_local_filesystem()->file_size(file_name, &file_size);
+ EXPECT_EQ(file_len, file_size);
+ }
+ // add 2 block
+ {
+ PBlock pblock;
+ generate_block(pblock, 1024);
+ file_len += pblock.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+
+ PBlock pblock1;
+ generate_block(pblock1, 2048);
+ file_len += pblock1.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+
+ EXPECT_EQ(Status::OK(), wal_writer.append_blocks(std::vector<PBlock*> {&pblock, &pblock1}));
+ io::global_local_filesystem()->file_size(file_name, &file_size);
+ EXPECT_EQ(file_len, file_size);
+ }
+ wal_writer.finalize();
+ // read block
+ auto wal_reader = WalReader(file_name);
+ wal_reader.init();
+ auto block_count = 0;
+ while (true) {
+ doris::PBlock pblock;
+ Status st = wal_reader.read_block(pblock);
+ EXPECT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ if (st.ok()) {
+ ++block_count;
+ } else if (st.is<ErrorCode::END_OF_FILE>()) {
+ break;
+ }
+ vectorized::Block block;
Review Comment:
warning: variable 'block' is not initialized [cppcoreguidelines-init-variables]
```suggestion
vectorized::Block block = 0;
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,137 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/object_pool.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gmock/gmock.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "service/brpc.h"
+#include "testutil/test_util.h"
+#include "util/proto_util.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+#include "vec/runtime/vdata_stream_recvr.h"
+
+using ::testing::_;
+using ::testing::Return;
+using ::testing::SetArgPointee;
+using std::string;
+
+namespace doris {
+
+class WalReaderWriterTest : public testing::Test {
+public:
+ // create a mock cgroup folder
+ virtual void SetUp() { io::global_local_filesystem()->create_directory(_s_test_data_path); }
+
+ // delete the mock cgroup folder
+ virtual void TearDown() { io::global_local_filesystem()->delete_directory(_s_test_data_path); }
+
+ static std::string _s_test_data_path;
+};
+
+std::string WalReaderWriterTest::_s_test_data_path = "./log/wal_reader_writer_test";
+size_t block_rows = 1024;
+
+void covert_block_to_pb(
+ const vectorized::Block& block, PBlock* pblock,
+ segment_v2::CompressionTypePB compression_type = segment_v2::CompressionTypePB::SNAPPY) {
+ size_t uncompressed_bytes = 0;
+ size_t compressed_bytes = 0;
+ Status st = block.serialize(BeExecVersionManager::get_newest_version(), pblock,
+ &uncompressed_bytes, &compressed_bytes, compression_type);
+ EXPECT_TRUE(st.ok());
+ EXPECT_TRUE(uncompressed_bytes >= compressed_bytes);
+ EXPECT_EQ(compressed_bytes, pblock->column_values().size());
+
+ const vectorized::ColumnWithTypeAndName& type_and_name =
+ block.get_columns_with_type_and_name()[0];
+ EXPECT_EQ(type_and_name.name, pblock->column_metas()[0].name());
+}
+
+void generate_block(PBlock& pblock, int row_index) {
+ auto vec = vectorized::ColumnVector<int32_t>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < block_rows; ++i) {
+ data.push_back(i + row_index);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ vectorized::Block block({type_and_name});
+ covert_block_to_pb(block, &pblock, segment_v2::CompressionTypePB::SNAPPY);
+}
+
+TEST_F(WalReaderWriterTest, TestWriteAndRead1) {
+ std::string file_name = _s_test_data_path + "/abcd123.txt";
+ auto wal_writer = WalWriter(file_name);
+ wal_writer.init();
+ size_t file_len = 0;
+ int64_t file_size = -1;
+ // add 1 block
+ {
+ PBlock pblock;
+ generate_block(pblock, 0);
+
+ EXPECT_EQ(Status::OK(), wal_writer.append_blocks(std::vector<PBlock*> {&pblock}));
+ file_len += pblock.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+ io::global_local_filesystem()->file_size(file_name, &file_size);
+ EXPECT_EQ(file_len, file_size);
+ }
+ // add 2 block
+ {
+ PBlock pblock;
Review Comment:
warning: variable 'pblock' is not initialized [cppcoreguidelines-init-variables]
```suggestion
PBlock pblock = 0;
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,137 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/object_pool.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gmock/gmock.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "service/brpc.h"
+#include "testutil/test_util.h"
+#include "util/proto_util.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+#include "vec/runtime/vdata_stream_recvr.h"
+
+using ::testing::_;
+using ::testing::Return;
+using ::testing::SetArgPointee;
+using std::string;
+
+namespace doris {
+
+class WalReaderWriterTest : public testing::Test {
+public:
+ // create a mock cgroup folder
+ virtual void SetUp() { io::global_local_filesystem()->create_directory(_s_test_data_path); }
+
+ // delete the mock cgroup folder
+ virtual void TearDown() { io::global_local_filesystem()->delete_directory(_s_test_data_path); }
+
+ static std::string _s_test_data_path;
+};
+
+std::string WalReaderWriterTest::_s_test_data_path = "./log/wal_reader_writer_test";
+size_t block_rows = 1024;
+
+void covert_block_to_pb(
+ const vectorized::Block& block, PBlock* pblock,
+ segment_v2::CompressionTypePB compression_type = segment_v2::CompressionTypePB::SNAPPY) {
+ size_t uncompressed_bytes = 0;
+ size_t compressed_bytes = 0;
+ Status st = block.serialize(BeExecVersionManager::get_newest_version(), pblock,
+ &uncompressed_bytes, &compressed_bytes, compression_type);
+ EXPECT_TRUE(st.ok());
+ EXPECT_TRUE(uncompressed_bytes >= compressed_bytes);
+ EXPECT_EQ(compressed_bytes, pblock->column_values().size());
+
+ const vectorized::ColumnWithTypeAndName& type_and_name =
+ block.get_columns_with_type_and_name()[0];
+ EXPECT_EQ(type_and_name.name, pblock->column_metas()[0].name());
+}
+
+void generate_block(PBlock& pblock, int row_index) {
+ auto vec = vectorized::ColumnVector<int32_t>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < block_rows; ++i) {
+ data.push_back(i + row_index);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ vectorized::Block block({type_and_name});
+ covert_block_to_pb(block, &pblock, segment_v2::CompressionTypePB::SNAPPY);
+}
+
+TEST_F(WalReaderWriterTest, TestWriteAndRead1) {
+ std::string file_name = _s_test_data_path + "/abcd123.txt";
+ auto wal_writer = WalWriter(file_name);
+ wal_writer.init();
+ size_t file_len = 0;
+ int64_t file_size = -1;
+ // add 1 block
+ {
+ PBlock pblock;
+ generate_block(pblock, 0);
+
+ EXPECT_EQ(Status::OK(), wal_writer.append_blocks(std::vector<PBlock*> {&pblock}));
+ file_len += pblock.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+ io::global_local_filesystem()->file_size(file_name, &file_size);
+ EXPECT_EQ(file_len, file_size);
+ }
+ // add 2 block
+ {
+ PBlock pblock;
+ generate_block(pblock, 1024);
+ file_len += pblock.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+
+ PBlock pblock1;
+ generate_block(pblock1, 2048);
+ file_len += pblock1.ByteSizeLong() + WalWriter::LENGTH_SIZE + WalWriter::CHECKSUM_SIZE;
+
+ EXPECT_EQ(Status::OK(), wal_writer.append_blocks(std::vector<PBlock*> {&pblock, &pblock1}));
+ io::global_local_filesystem()->file_size(file_name, &file_size);
+ EXPECT_EQ(file_len, file_size);
+ }
+ wal_writer.finalize();
+ // read block
+ auto wal_reader = WalReader(file_name);
+ wal_reader.init();
+ auto block_count = 0;
+ while (true) {
+ doris::PBlock pblock;
+ Status st = wal_reader.read_block(pblock);
Review Comment:
warning: variable 'st' is not initialized [cppcoreguidelines-init-variables]
```suggestion
Status st = 0 = wal_reader.read_block(pblock);
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <gtest/gtest.h>
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+extern std::string k_response;
+extern std::string k_request_line;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = WalManager::create_shared(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_response = "OK";
+ k_request_line = "\"Status\": \"Success\",\n";
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
+ std::string wal_101 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
+ createWal(wal_100);
+ createWal(wal_101);
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_2_id);
+ std::string wal_200 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_200_id;
+ std::string wal_201 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_201_id;
+ createWal(wal_200);
+ createWal(wal_201);
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size(tb_1_id) > 0 ||
+ _env->_wal_manager->get_wal_table_size(tb_2_id) > 0) {
+ sleep(1);
+ continue;
+ }
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+ ASSERT_TRUE(!std::filesystem::exists(wal_101));
+ ASSERT_TRUE(!std::filesystem::exists(wal_200));
+ ASSERT_TRUE(!std::filesystem::exists(wal_201));
+}
+
+TEST_F(WalManagerTest, not_need_recovery) {
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
+ createWal(wal_100);
+
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size("1") > 0) {
+ sleep(1);
+ continue;
+ }
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+}
+
+TEST_F(WalManagerTest, recover_fail) {
+ k_request_line = "\"Status\": \"Fail\",\n";
+ k_check_wal_result.__set_need_recovery(true);
+ config::group_commit_replay_wal_retry_num = 3;
+
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
Review Comment:
warning: variable 'wal_100' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_100 = 0 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
```
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,137 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/object_pool.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gmock/gmock.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "service/brpc.h"
+#include "testutil/test_util.h"
+#include "util/proto_util.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+#include "vec/runtime/vdata_stream_recvr.h"
+
+using ::testing::_;
+using ::testing::Return;
+using ::testing::SetArgPointee;
+using std::string;
+
+namespace doris {
+
+class WalReaderWriterTest : public testing::Test {
+public:
+ // create a mock cgroup folder
+ virtual void SetUp() { io::global_local_filesystem()->create_directory(_s_test_data_path); }
+
+ // delete the mock cgroup folder
+ virtual void TearDown() { io::global_local_filesystem()->delete_directory(_s_test_data_path); }
+
+ static std::string _s_test_data_path;
+};
+
+std::string WalReaderWriterTest::_s_test_data_path = "./log/wal_reader_writer_test";
+size_t block_rows = 1024;
+
+void covert_block_to_pb(
+ const vectorized::Block& block, PBlock* pblock,
+ segment_v2::CompressionTypePB compression_type = segment_v2::CompressionTypePB::SNAPPY) {
+ size_t uncompressed_bytes = 0;
+ size_t compressed_bytes = 0;
+ Status st = block.serialize(BeExecVersionManager::get_newest_version(), pblock,
+ &uncompressed_bytes, &compressed_bytes, compression_type);
+ EXPECT_TRUE(st.ok());
+ EXPECT_TRUE(uncompressed_bytes >= compressed_bytes);
+ EXPECT_EQ(compressed_bytes, pblock->column_values().size());
+
+ const vectorized::ColumnWithTypeAndName& type_and_name =
+ block.get_columns_with_type_and_name()[0];
+ EXPECT_EQ(type_and_name.name, pblock->column_metas()[0].name());
+}
+
+void generate_block(PBlock& pblock, int row_index) {
+ auto vec = vectorized::ColumnVector<int32_t>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < block_rows; ++i) {
+ data.push_back(i + row_index);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ vectorized::Block block({type_and_name});
+ covert_block_to_pb(block, &pblock, segment_v2::CompressionTypePB::SNAPPY);
+}
+
+TEST_F(WalReaderWriterTest, TestWriteAndRead1) {
+ std::string file_name = _s_test_data_path + "/abcd123.txt";
+ auto wal_writer = WalWriter(file_name);
+ wal_writer.init();
+ size_t file_len = 0;
+ int64_t file_size = -1;
+ // add 1 block
+ {
+ PBlock pblock;
Review Comment:
warning: variable 'pblock' is not initialized [cppcoreguidelines-init-variables]
```suggestion
PBlock pblock = 0;
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include "olap/wal_manager.h"
+
+#include <gtest/gtest.h>
+
+#include <filesystem>
+#include <map>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "io/fs/local_file_system.h"
+#include "runtime/decimalv2_value.h"
+#include "runtime/exec_env.h"
+#include "runtime/result_queue_mgr.h"
+#include "runtime/runtime_state.h"
+#include "runtime/stream_load/new_load_stream_mgr.h"
+#include "runtime/types.h"
+#include "service/brpc.h"
+#include "util/brpc_client_cache.h"
+#include "util/cpu_info.h"
+#include "util/debug/leakcheck_disabler.h"
+#include "util/proto_util.h"
+
+namespace doris {
+
+extern TCheckWalResult k_check_wal_result;
+extern TLoadTxnBeginResult k_stream_load_begin_result;
+extern Status k_stream_load_plan_status;
+extern std::string k_response;
+extern std::string k_request_line;
+
+class WalManagerTest : public testing::Test {
+public:
+ WalManagerTest() {}
+ virtual ~WalManagerTest() {}
+ void SetUp() override {
+ prepare();
+ _env = ExecEnv::GetInstance();
+ _env->_master_info = new TMasterInfo();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = 1234;
+ _env->new_load_stream_mgr() = NewLoadStreamMgr::create_shared();
+ _env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
+ _env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
+ _env->_stream_load_executor = StreamLoadExecutor::create_shared(_env);
+ _env->_wal_manager = WalManager::create_shared(_env, wal_dir);
+ k_check_wal_result = TCheckWalResult();
+ k_stream_load_begin_result = TLoadTxnBeginResult();
+ k_stream_load_plan_status = Status::OK();
+ }
+ void TearDown() override {
+ io::global_local_filesystem()->delete_directory(wal_dir);
+ SAFE_DELETE(_env->_function_client_cache);
+ SAFE_DELETE(_env->_internal_client_cache);
+ SAFE_DELETE(_env->_master_info);
+ }
+
+ void prepare() { io::global_local_filesystem()->create_directory(wal_dir); }
+
+ void createWal(const std::string& wal_path) {
+ auto wal_writer = WalWriter(wal_path);
+ wal_writer.init();
+ wal_writer.finalize();
+ }
+
+private:
+ ExecEnv* _env = nullptr;
+ std::string wal_dir = "./wal_test";
+ std::string tmp_dir = "./wal_test/tmp";
+};
+
+TEST_F(WalManagerTest, recovery_normal) {
+ k_response = "OK";
+ k_request_line = "\"Status\": \"Success\",\n";
+ k_check_wal_result.__set_need_recovery(true);
+
+ std::string db_id = "1";
+ std::string tb_1_id = "1";
+ std::string wal_100_id = "100";
+ std::string wal_101_id = "101";
+ std::string tb_2_id = "2";
+ std::string wal_200_id = "200";
+ std::string wal_201_id = "201";
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_1_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_100_id;
+ std::string wal_101 = wal_dir + "/" + db_id + "/" + tb_1_id + "/" + wal_101_id;
+ createWal(wal_100);
+ createWal(wal_101);
+
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_2_id);
+ std::string wal_200 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_200_id;
+ std::string wal_201 = wal_dir + "/" + db_id + "/" + tb_2_id + "/" + wal_201_id;
+ createWal(wal_200);
+ createWal(wal_201);
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size(tb_1_id) > 0 ||
+ _env->_wal_manager->get_wal_table_size(tb_2_id) > 0) {
+ sleep(1);
+ continue;
+ }
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+ ASSERT_TRUE(!std::filesystem::exists(wal_101));
+ ASSERT_TRUE(!std::filesystem::exists(wal_200));
+ ASSERT_TRUE(!std::filesystem::exists(wal_201));
+}
+
+TEST_F(WalManagerTest, not_need_recovery) {
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
+ createWal(wal_100);
+
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size("1") > 0) {
+ sleep(1);
+ continue;
+ }
+ ASSERT_TRUE(!std::filesystem::exists(wal_100));
+}
+
+TEST_F(WalManagerTest, recover_fail) {
+ k_request_line = "\"Status\": \"Fail\",\n";
+ k_check_wal_result.__set_need_recovery(true);
+ config::group_commit_replay_wal_retry_num = 3;
+
+ std::string db_id = "1";
+ std::string tb_id = "1";
+ std::string wal_id = "100";
+ std::filesystem::create_directory(wal_dir + "/" + db_id);
+ std::filesystem::create_directory(wal_dir + "/" + db_id + "/" + tb_id);
+ std::string wal_100 = wal_dir + "/" + db_id + "/" + tb_id + "/" + wal_id;
+ createWal(wal_100);
+
+ _env->wal_mgr()->init();
+
+ while (_env->_wal_manager->get_wal_table_size("1") > 0) {
+ sleep(1);
+ continue;
+ }
+ std::string tmp_file = tmp_dir + "/" + db_id + "_" + tb_id + "_" + wal_id;
Review Comment:
warning: variable 'tmp_file' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string tmp_file = 0 = tmp_dir + "/" + db_id + "_" + tb_id + "_" + wal_id;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1715770134
TeamCity be ut coverage result:
Function Coverage: 37.42% (8052/21520)
Line Coverage: 29.34% (64759/220712)
Region Coverage: 28.30% (33661/118961)
Branch Coverage: 24.77% (17246/69630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7a81ad137e16e909c8afb3bbd93ef20b2bf6b2c6_7a81ad137e16e909c8afb3bbd93ef20b2bf6b2c6/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1715857684
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.17 seconds
stream load tsv: 607 seconds loaded 74807831229 Bytes, about 117 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162331987 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1716230899
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.38 seconds
stream load tsv: 599 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162394917 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1716903252
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.65 seconds
stream load tsv: 608 seconds loaded 74807831229 Bytes, about 117 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162355957 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1716919856
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1715631455
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1734918738
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1335173798
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -308,9 +311,25 @@ Status GroupCommitTable::_finish_group_commit_load(int64_t db_id, int64_t table_
<< ", instance_id=" << print_id(instance_id)
<< ", executor status=" << status.to_string()
<< ", request commit status=" << st.to_string();
+ if (!prepare_failed) {
+ _exec_env->wal_mgr()->add_wal_path(_db_id, table_id, txn_id, label);
+ std::string wal_path;
+ _exec_env->wal_mgr()->get_wal_path(txn_id, wal_path);
+ _exec_env->wal_mgr()->add_recover_wal(std::to_string(db_id), std::to_string(table_id),
+ std::vector<std::string> {wal_path});
Review Comment:
Do we need handle return value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1731955798
TeamCity be ut coverage result:
Function Coverage: 36.38% (8062/22162)
Line Coverage: 28.73% (64882/225858)
Region Coverage: 27.60% (33673/122009)
Branch Coverage: 24.25% (17214/70982)
Coverage Report: http://coverage.selectdb-in.cc/coverage/395634fab6a82e745254835a0bcfc81e4dd6f1ec_395634fab6a82e745254835a0bcfc81e4dd6f1ec/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1297893777
##########
be/src/runtime/exec_env_init.cpp:
##########
@@ -170,6 +171,7 @@ Status ExecEnv::_init(const std::vector<StorePath>& store_paths) {
_block_spill_mgr = new BlockSpillManager(_store_paths);
_file_meta_cache = new FileMetaCache(config::max_external_file_meta_cache_num);
_memtable_memory_limiter = std::make_unique<MemTableMemoryLimiter>();
+ _wal_manager = new WalManager(this, config::group_commit_replay_wal_dir);
Review Comment:
use smart pointer, not use raw pointer.
Add ENABLE_FACTORY_CREATOR to all classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1314391103
##########
gensrc/thrift/FrontendService.thrift:
##########
@@ -644,6 +647,29 @@ struct TStreamLoadWithLoadStatusResult {
6: optional i64 unselected_rows
}
+struct TCheckWalRequest {
+ 1: optional i64 table_id
+ 2: optional i64 wal_id
+ 3: optional i64 db_id
+}
+
+struct TCheckWalResult {
+ 1: optional Status.TStatus status
+ 2: optional bool need_recovery
+}
+
+struct TUpdateWalMapRequest {
Review Comment:
TCheckWalRequest and TUpdateWalMapRequest are not used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1712116927
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.41 seconds
stream load tsv: 544 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162100002 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1712394012
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1334703244
##########
be/test/olap/wal_reader_writer_test.cpp:
##########
@@ -0,0 +1,137 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include <filesystem>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/object_pool.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gmock/gmock.h"
+#include "io/fs/local_file_system.h"
+#include "olap/wal_reader.h"
+#include "olap/wal_writer.h"
+#include "runtime/exec_env.h"
+#include "service/brpc.h"
+#include "testutil/test_util.h"
+#include "util/proto_util.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+#include "vec/runtime/vdata_stream_recvr.h"
+
+using ::testing::_;
+using ::testing::Return;
+using ::testing::SetArgPointee;
+using std::string;
+
+namespace doris {
+
+class WalReaderWriterTest : public testing::Test {
+public:
+ // create a mock cgroup folder
+ virtual void SetUp() { io::global_local_filesystem()->create_directory(_s_test_data_path); }
+
+ // delete the mock cgroup folder
+ virtual void TearDown() { io::global_local_filesystem()->delete_directory(_s_test_data_path); }
+
+ static std::string _s_test_data_path;
+};
+
+std::string WalReaderWriterTest::_s_test_data_path = "./log/wal_reader_writer_test";
+size_t block_rows = 1024;
+
+void covert_block_to_pb(
+ const vectorized::Block& block, PBlock* pblock,
+ segment_v2::CompressionTypePB compression_type = segment_v2::CompressionTypePB::SNAPPY) {
+ size_t uncompressed_bytes = 0;
+ size_t compressed_bytes = 0;
+ Status st = block.serialize(BeExecVersionManager::get_newest_version(), pblock,
+ &uncompressed_bytes, &compressed_bytes, compression_type);
+ EXPECT_TRUE(st.ok());
+ EXPECT_TRUE(uncompressed_bytes >= compressed_bytes);
+ EXPECT_EQ(compressed_bytes, pblock->column_values().size());
+
+ const vectorized::ColumnWithTypeAndName& type_and_name =
+ block.get_columns_with_type_and_name()[0];
+ EXPECT_EQ(type_and_name.name, pblock->column_metas()[0].name());
+}
+
+void generate_block(PBlock& pblock, int row_index) {
+ auto vec = vectorized::ColumnVector<int32_t>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < block_rows; ++i) {
+ data.push_back(i + row_index);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ vectorized::Block block({type_and_name});
+ covert_block_to_pb(block, &pblock, segment_v2::CompressionTypePB::SNAPPY);
+}
+
+TEST_F(WalReaderWriterTest, TestWriteAndRead1) {
+ std::string file_name = _s_test_data_path + "/abcd123.txt";
Review Comment:
warning: variable 'file_name' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string file_name = 0 = _s_test_data_path + "/abcd123.txt";
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
Review Comment:
warning: variable 'query_options' is not initialized [cppcoreguidelines-init-variables]
```suggestion
TQueryOptions query_options = 0;
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
Review Comment:
warning: variable 'fragment_id' is not initialized [cppcoreguidelines-init-variables]
```suggestion
TUniqueId fragment_id = 0;
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
Review Comment:
warning: variable 'options' is not initialized [cppcoreguidelines-init-variables]
```suggestion
brpc::ServerOptions options = 0;
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
Review Comment:
warning: variable 'tdesc_tbl' is not initialized [cppcoreguidelines-init-variables]
```suggestion
TDescriptorTable tdesc_tbl = 0;
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
Review Comment:
warning: variable 'srcRow' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string srcRow = 0 = org_block.dump_one_line(i, org_block.columns());
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
+ ASSERT_TRUE(std::strcmp(srcRow.c_str(), walRow.c_str()) == 0);
+ }
+}
+
+TEST_F(VOlapTableSinkTest, group_commit_with_filter_row) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
Review Comment:
warning: variable 'tdesc_tbl' is not initialized [cppcoreguidelines-init-variables]
```suggestion
TDescriptorTable tdesc_tbl = 0;
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
+ ASSERT_TRUE(std::strcmp(srcRow.c_str(), walRow.c_str()) == 0);
+ }
+}
+
+TEST_F(VOlapTableSinkTest, group_commit_with_filter_row) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
Review Comment:
warning: variable 'query_options' is not initialized [cppcoreguidelines-init-variables]
```suggestion
TQueryOptions query_options = 0;
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
+ ASSERT_TRUE(std::strcmp(srcRow.c_str(), walRow.c_str()) == 0);
+ }
+}
+
+TEST_F(VOlapTableSinkTest, group_commit_with_filter_row) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
Review Comment:
warning: variable 'output_set' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::set<std::string> output_set = 0;
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
+ ASSERT_TRUE(std::strcmp(srcRow.c_str(), walRow.c_str()) == 0);
+ }
+}
+
+TEST_F(VOlapTableSinkTest, group_commit_with_filter_row) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("abcde1234567890", 15);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 2, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(1, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
Review Comment:
warning: variable 'wal_path' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_path = 0 = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
+ ASSERT_TRUE(std::strcmp(srcRow.c_str(), walRow.c_str()) == 0);
+ }
+}
+
+TEST_F(VOlapTableSinkTest, group_commit_with_filter_row) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("abcde1234567890", 15);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 2, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(1, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows() - 1, wal_block.rows());
+ for (int i = 0; i < wal_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
Review Comment:
warning: variable 'srcRow' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string srcRow = 0 = org_block.dump_one_line(i, org_block.columns());
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
+ ASSERT_TRUE(std::strcmp(srcRow.c_str(), walRow.c_str()) == 0);
+ }
+}
+
+TEST_F(VOlapTableSinkTest, group_commit_with_filter_row) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
Review Comment:
warning: variable 'options' is not initialized [cppcoreguidelines-init-variables]
```suggestion
brpc::ServerOptions options = 0;
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
Review Comment:
warning: variable 'column_ptr' is not initialized [cppcoreguidelines-init-variables]
```suggestion
auto* column_ptr = nullptr = columns[col_idx++].get();
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
Review Comment:
warning: variable 'output_set' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::set<std::string> output_set = 0;
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
Review Comment:
warning: variable 'wal_path' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string wal_path = 0 = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
+ ASSERT_TRUE(std::strcmp(srcRow.c_str(), walRow.c_str()) == 0);
+ }
+}
+
+TEST_F(VOlapTableSinkTest, group_commit_with_filter_row) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
Review Comment:
warning: variable 'fragment_id' is not initialized [cppcoreguidelines-init-variables]
```suggestion
TUniqueId fragment_id = 0;
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
Review Comment:
warning: variable 'walRow' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string walRow = 0 = wal_block.dump_one_line(i, org_block.columns());
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
+ ASSERT_TRUE(std::strcmp(srcRow.c_str(), walRow.c_str()) == 0);
+ }
+}
+
+TEST_F(VOlapTableSinkTest, group_commit_with_filter_row) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("abcde1234567890", 15);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 2, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(1, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows() - 1, wal_block.rows());
+ for (int i = 0; i < wal_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
Review Comment:
warning: variable 'walRow' is not initialized [cppcoreguidelines-init-variables]
```suggestion
std::string walRow = 0 = wal_block.dump_one_line(i, org_block.columns());
```
##########
be/test/vec/exec/vtablet_sink_test.cpp:
##########
@@ -835,5 +844,259 @@ TEST_F(VOlapTableSinkTest, decimal) {
ASSERT_TRUE(output_set.count("(12, 12.300000000)") > 0);
ASSERT_TRUE(output_set.count("(13, 123.120000000)") > 0);
}
+
+TEST_F(VOlapTableSinkTest, group_commit) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
+ auto column_vector_int = column_ptr;
+ int int_val = 12;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 13;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+ int_val = 14;
+ column_vector_int->insert_data((const char*)&int_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_bigint = column_ptr;
+ int64_t int64_val = 9;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 25;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+ int64_val = 50;
+ column_vector_bigint->insert_data((const char*)&int64_val, 0);
+
+ column_ptr = columns[col_idx++].get();
+ auto column_vector_str = column_ptr;
+ column_vector_str->insert_data("abc", 3);
+ column_vector_str->insert_data("abcd", 4);
+ column_vector_str->insert_data("1234567890", 10);
+
+ vectorized::Block block;
+ col_idx = 0;
+ for (const auto slot_desc : tuple_desc->slots()) {
+ block.insert(vectorized::ColumnWithTypeAndName(std::move(columns[col_idx++]),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name()));
+ }
+ vectorized::Block org_block(block);
+
+ // send
+ st = sink.send(&state, &block);
+ ASSERT_TRUE(st.ok());
+ // close
+ st = sink.close(&state, Status::OK());
+ ASSERT_TRUE(st.ok() || st.to_string() == "Internal error: wait close failed. ")
+ << st.to_string();
+
+ // each node has a eof
+ ASSERT_EQ(2, service->_eof_counters);
+ ASSERT_EQ(2 * 3, service->_row_counters);
+
+ // 2node * 2
+ ASSERT_EQ(0, state.num_rows_load_filtered());
+
+ std::string wal_path = wal_dir + "/" + std::to_string(t_data_sink.olap_table_sink.db_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.table_id) + "/" +
+ std::to_string(t_data_sink.olap_table_sink.txn_id) + "_" +
+ state.import_label();
+ doris::PBlock pblock;
+ auto wal_reader = WalReader(wal_path);
+ st = wal_reader.init();
+ ASSERT_TRUE(st.ok());
+ st = wal_reader.read_block(pblock);
+ ASSERT_TRUE(st.ok());
+ vectorized::Block wal_block;
+ wal_block.deserialize(pblock);
+ ASSERT_TRUE(st.ok() || st.is<ErrorCode::END_OF_FILE>());
+ ASSERT_EQ(org_block.rows(), wal_block.rows());
+ for (int i = 0; i < org_block.rows(); i++) {
+ std::string srcRow = org_block.dump_one_line(i, org_block.columns());
+ std::string walRow = wal_block.dump_one_line(i, org_block.columns());
+ ASSERT_TRUE(std::strcmp(srcRow.c_str(), walRow.c_str()) == 0);
+ }
+}
+
+TEST_F(VOlapTableSinkTest, group_commit_with_filter_row) {
+ // start brpc service first
+ _server = new brpc::Server();
+ auto service = new VTestInternalService();
+ ASSERT_EQ(_server->AddService(service, brpc::SERVER_OWNS_SERVICE), 0);
+ brpc::ServerOptions options;
+ {
+ debug::ScopedLeakCheckDisabler disable_lsan;
+ _server->Start(4356, &options);
+ }
+
+ TUniqueId fragment_id;
+ TQueryOptions query_options;
+ query_options.batch_size = 1;
+ query_options.be_exec_version = 0;
+ RuntimeState state(fragment_id, query_options, TQueryGlobals(), _env);
+ state.init_mem_trackers(TUniqueId());
+
+ ObjectPool obj_pool;
+ TDescriptorTable tdesc_tbl;
+ auto t_data_sink = get_data_sink(&tdesc_tbl);
+
+ // crate desc_tabl
+ DescriptorTbl* desc_tbl = nullptr;
+ auto st = DescriptorTbl::create(&obj_pool, tdesc_tbl, &desc_tbl);
+ ASSERT_TRUE(st.ok());
+ state._desc_tbl = desc_tbl;
+ state._wal_id = 789;
+ state._import_label = "test";
+
+ TupleDescriptor* tuple_desc = desc_tbl->get_tuple_descriptor(0);
+ LOG(INFO) << "tuple_desc=" << tuple_desc->debug_string();
+
+ RowDescriptor row_desc(*desc_tbl, {0}, {false});
+ service->_row_desc = &row_desc;
+ std::set<std::string> output_set;
+ service->_output_set = &output_set;
+
+ std::vector<TExpr> exprs;
+ VOlapTableSink sink(&obj_pool, row_desc, exprs, true);
+
+ // init
+ st = sink.init(t_data_sink);
+ ASSERT_TRUE(st.ok());
+ // prepare
+ st = sink.prepare(&state);
+ ASSERT_TRUE(st.ok());
+ // open
+ st = sink.open(&state);
+ ASSERT_TRUE(st.ok());
+
+ int slot_count = tuple_desc->slots().size();
+ std::vector<vectorized::MutableColumnPtr> columns(slot_count);
+ for (int i = 0; i < slot_count; i++) {
+ columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
+ }
+
+ int col_idx = 0;
+ auto* column_ptr = columns[col_idx++].get();
Review Comment:
warning: variable 'column_ptr' is not initialized [cppcoreguidelines-init-variables]
```suggestion
auto* column_ptr = nullptr = columns[col_idx++].get();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1733057660
TeamCity be ut coverage result:
Function Coverage: 36.27% (8063/22232)
Line Coverage: 28.62% (64891/226758)
Region Coverage: 27.47% (33671/122590)
Branch Coverage: 24.14% (17217/71310)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3338c19e5ead32bed6936c6ca3ccb8cd868a4c66_3338c19e5ead32bed6936c6ca3ccb8cd868a4c66/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1732979986
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.51 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162108471 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1713175018
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.09 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162474714 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1716971965
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.65 seconds
stream load tsv: 591 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162423334 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #23053:
URL: https://github.com/apache/doris/pull/23053#issuecomment-1718516742
TeamCity be ut coverage result:
Function Coverage: 37.44% (8089/21607)
Line Coverage: 29.35% (64940/221265)
Region Coverage: 28.30% (33736/119210)
Branch Coverage: 24.74% (17255/69750)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ab382df4188d981e16bb296c196a74d532857c73_ab382df4188d981e16bb296c196a74d532857c73/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mymeiyi commented on a diff in pull request #23053: [feature](insert) Support wal for group commit insert
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #23053:
URL: https://github.com/apache/doris/pull/23053#discussion_r1322299146
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -2020,6 +2025,39 @@ private void httpStreamPutImpl(TStreamLoadPutRequest request, TStreamLoadPutResu
}
}
+ @Override
+ public TCheckWalResult checkWal(TCheckWalRequest request) {
+ TCheckWalResult result = new TCheckWalResult();
+ TStatus status = new TStatus(TStatusCode.OK);
+ result.setStatus(status);
+ boolean res = true;
+ try {
+ res = checkWalImpl(request.getDbId(), request.getWalId(), request.getWalId());
+ } catch (Throwable e) {
+ LOG.warn("catch unknown result.", e);
+ status.setStatusCode(TStatusCode.INTERNAL_ERROR);
+ status.addToErrorMsgs(e.getClass().getSimpleName() + ": " + Strings.nullToEmpty(e.getMessage()));
+ return result;
+ }
+ result.setNeedRecovery(res);
+ return result;
+ }
+
+ private boolean checkWalImpl(long dbId, long walId, long transactionId) {
+ TransactionState state = Env.getCurrentGlobalTransactionMgr()
Review Comment:
Here we should check all label related txns, not only the first txn id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org