You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Yukang-Lian (via GitHub)" <gi...@apache.org> on 2024/07/15 08:57:01 UTC
[PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Yukang-Lian opened a new pull request, #37824:
URL: https://github.com/apache/doris/pull/37824
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
Pick #34871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37824:
URL: https://github.com/apache/doris/pull/37824#issuecomment-2228067333
<details>
<summary>TPC-H: <b>Total hot run time: 50190 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e4eef77f951087f35bf6184618e82a865ba093a9, data reload: false
------ Round 1 ----------------------------------
q1 17649 4392 4410 4392
q2 2071 162 148 148
q3 10264 1870 1959 1870
q4 10113 1269 1346 1269
q5 8575 3954 3930 3930
q6 261 125 126 125
q7 2106 1608 1632 1608
q8 9542 2766 2725 2725
q9 14291 10520 10560 10520
q10 8640 3531 3510 3510
q11 424 248 255 248
q12 473 307 306 306
q13 18375 3933 4054 3933
q14 363 328 334 328
q15 506 466 463 463
q16 661 573 564 564
q17 1142 980 966 966
q18 7354 6794 6826 6794
q19 1808 1669 1619 1619
q20 519 310 311 310
q21 4431 4124 4130 4124
q22 513 438 439 438
Total cold run time: 120081 ms
Total hot run time: 50190 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4310 4301 4483 4301
q2 321 227 232 227
q3 4194 4150 4130 4130
q4 2759 2745 2752 2745
q5 7195 7160 7144 7144
q6 239 121 120 120
q7 3258 2829 2792 2792
q8 4375 4475 4482 4475
q9 17404 17183 17100 17100
q10 4211 4300 4281 4281
q11 762 703 688 688
q12 1032 850 856 850
q13 7216 3732 3754 3732
q14 464 433 421 421
q15 515 453 457 453
q16 736 694 678 678
q17 3773 3887 3866 3866
q18 8815 8686 8713 8686
q19 1764 1673 1693 1673
q20 2384 2149 2081 2081
q21 8470 8592 8426 8426
q22 1066 1008 992 992
Total cold run time: 85263 ms
Total hot run time: 79861 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian closed pull request #37824: [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)"
URL: https://github.com/apache/doris/pull/37824
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37824:
URL: https://github.com/apache/doris/pull/37824#issuecomment-2228108266
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit e4eef77f951087f35bf6184618e82a865ba093a9 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 21.7 seconds inserted 10000000 Rows, about 460K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #37824:
URL: https://github.com/apache/doris/pull/37824#issuecomment-2228008474
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #37824:
URL: https://github.com/apache/doris/pull/37824#issuecomment-2228013383
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37824:
URL: https://github.com/apache/doris/pull/37824#issuecomment-2228008051
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37824:
URL: https://github.com/apache/doris/pull/37824#issuecomment-2228102450
<details>
<summary>ClickBench: <b>Total hot run time: 30.92 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e4eef77f951087f35bf6184618e82a865ba093a9, data reload: false
query1 0.02 0.03 0.02
query2 0.07 0.02 0.02
query3 0.25 0.05 0.04
query4 1.79 0.06 0.06
query5 0.53 0.53 0.51
query6 1.28 0.67 0.63
query7 0.02 0.01 0.01
query8 0.03 0.03 0.02
query9 0.52 0.49 0.48
query10 0.54 0.56 0.53
query11 0.12 0.09 0.09
query12 0.11 0.08 0.08
query13 0.63 0.62 0.60
query14 0.78 0.77 0.78
query15 0.78 0.76 0.75
query16 0.36 0.38 0.37
query17 0.99 1.02 1.01
query18 0.24 0.24 0.24
query19 1.94 1.83 1.76
query20 0.01 0.01 0.02
query21 15.47 0.57 0.56
query22 2.00 2.58 1.60
query23 16.50 1.13 0.93
query24 6.06 1.56 1.40
query25 0.39 0.12 0.05
query26 0.70 0.15 0.16
query27 0.03 0.04 0.05
query28 6.34 0.75 0.72
query29 12.65 2.26 2.13
query30 0.60 0.48 0.54
query31 2.82 0.38 0.38
query32 3.37 0.50 0.49
query33 3.06 3.08 3.07
query34 15.25 4.82 4.84
query35 4.86 4.85 4.85
query36 1.04 1.01 1.02
query37 0.06 0.04 0.04
query38 0.03 0.02 0.02
query39 0.02 0.01 0.01
query40 0.16 0.14 0.14
query41 0.07 0.01 0.02
query42 0.02 0.02 0.01
query43 0.02 0.01 0.02
Total cold run time: 102.53 s
Total hot run time: 30.92 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37824:
URL: https://github.com/apache/doris/pull/37824#issuecomment-2228072900
TeamCity be ut coverage result:
Function Coverage: 37.86% (8120/21448)
Line Coverage: 29.53% (66570/225413)
Region Coverage: 29.00% (34310/118290)
Branch Coverage: 24.88% (17629/70856)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e4eef77f951087f35bf6184618e82a865ba093a9_e4eef77f951087f35bf6184618e82a865ba093a9/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #37824:
URL: https://github.com/apache/doris/pull/37824#discussion_r1677516715
##########
be/src/olap/segment_loader.h:
##########
@@ -109,15 +109,20 @@ class SegmentLoader {
// Load segments of "rowset", return the "cache_handle" which contains segments.
// If use_cache is true, it will be loaded from _cache.
Status load_segments(const BetaRowsetSharedPtr& rowset, SegmentCacheHandle* cache_handle,
- bool use_cache = false);
+ bool use_cache = false, bool need_load_pk_index_and_bf = false);
void erase_segments(const SegmentCache::CacheKey& key);
+ // Just used for BE UT
+ int64_t cache_mem_usage() const { return _cache_mem_usage; }
Review Comment:
warning: function 'cache_mem_usage' should be marked [[nodiscard]] [modernize-use-nodiscard]
```suggestion
[[nodiscard]] int64_t cache_mem_usage() const { return _cache_mem_usage; }
```
##########
be/test/olap/segment_cache_test.cpp:
##########
@@ -0,0 +1,347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gen_cpp/AgentService_types.h>
+#include <gtest/gtest-message.h>
+#include <gtest/gtest-test-part.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include <iostream>
+#include <map>
+#include <memory>
+#include <mutex>
+#include <string>
+#include <utility>
+
+#include "common/config.h"
+#include "common/object_pool.h"
+#include "exec/tablet_info.h"
+#include "gen_cpp/Descriptors_types.h"
+#include "gen_cpp/Types_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gtest/gtest_pred_impl.h"
+#include "io/fs/local_file_system.h"
+#include "olap/data_dir.h"
+#include "olap/delta_writer.h"
+#include "olap/iterators.h"
+#include "olap/olap_define.h"
+#include "olap/options.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/segment_v2/segment.h"
+#include "olap/schema.h"
+#include "olap/segment_loader.h"
+#include "olap/storage_engine.h"
+#include "olap/tablet.h"
+#include "olap/tablet_manager.h"
+#include "olap/task/engine_publish_version_task.h"
+#include "olap/txn_manager.h"
+#include "runtime/define_primitive_type.h"
+#include "runtime/descriptor_helper.h"
+#include "runtime/descriptors.h"
+#include "runtime/exec_env.h"
+#include "vec/columns/column.h"
+#include "vec/core/block.h"
+#include "vec/core/column_with_type_and_name.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris {
+class OlapMeta;
+
+// This is DeltaWriter unit test which used by streaming load.
+// And also it should take schema change into account after streaming load.
+
+static const uint32_t MAX_PATH_LEN = 1024;
+
+static StorageEngine* k_engine = nullptr;
+
+static void set_up() {
+ char buffer[MAX_PATH_LEN];
+ EXPECT_NE(getcwd(buffer, MAX_PATH_LEN), nullptr);
+ config::storage_root_path = std::string(buffer) + "/data_test";
+ io::global_local_filesystem()->delete_and_create_directory(config::storage_root_path);
+ std::vector<StorePath> paths;
+ paths.emplace_back(config::storage_root_path, -1);
+
+ doris::EngineOptions options;
+ options.store_paths = paths;
+ Status s = doris::StorageEngine::open(options, &k_engine);
+ EXPECT_TRUE(s.ok()) << s.to_string();
+
+ ExecEnv* exec_env = doris::ExecEnv::GetInstance();
+ exec_env->set_storage_engine(k_engine);
+ k_engine->start_bg_threads();
+}
+
+static void tear_down() {
+ if (k_engine != nullptr) {
+ k_engine->stop();
+ delete k_engine;
+ k_engine = nullptr;
+ }
+ EXPECT_EQ(system("rm -rf ./data_test"), 0);
+ io::global_local_filesystem()->delete_directory(std::string(getenv("DORIS_HOME")) + "/" +
+ UNUSED_PREFIX);
+}
+
+static void create_tablet_request_with_sequence_col(int64_t tablet_id, int32_t schema_hash,
+ TCreateTabletReq* request,
+ bool enable_mow = false) {
+ request->tablet_id = tablet_id;
+ request->partition_id = 10000;
+ request->__set_version(1);
+ request->tablet_schema.schema_hash = schema_hash;
+ request->tablet_schema.short_key_column_count = 2;
+ request->tablet_schema.keys_type = TKeysType::UNIQUE_KEYS;
+ request->tablet_schema.storage_type = TStorageType::COLUMN;
+ request->tablet_schema.__set_sequence_col_idx(4);
+ request->__set_storage_format(TStorageFormat::V2);
+ request->__set_enable_unique_key_merge_on_write(enable_mow);
+
+ TColumn k1;
+ k1.column_name = "k1";
+ k1.__set_is_key(true);
+ k1.column_type.type = TPrimitiveType::TINYINT;
+ request->tablet_schema.columns.push_back(k1);
+
+ TColumn k2;
+ k2.column_name = "k2";
+ k2.__set_is_key(true);
+ k2.column_type.type = TPrimitiveType::SMALLINT;
+ request->tablet_schema.columns.push_back(k2);
+
+ TColumn v1;
+ v1.column_name = "v1";
+ v1.__set_is_key(false);
+ v1.column_type.type = TPrimitiveType::DATETIME;
+ v1.__set_aggregation_type(TAggregationType::REPLACE);
+ request->tablet_schema.columns.push_back(v1);
+
+ TColumn v2;
+ v2.column_name = "v2";
+ v2.__set_is_key(false);
+ v2.column_type.type = TPrimitiveType::DATEV2;
+ v2.__set_aggregation_type(TAggregationType::REPLACE);
+ request->tablet_schema.columns.push_back(v2);
+
+ TColumn sequence_col;
+ sequence_col.column_name = SEQUENCE_COL;
+ sequence_col.__set_is_key(false);
+ sequence_col.column_type.type = TPrimitiveType::INT;
+ sequence_col.__set_aggregation_type(TAggregationType::REPLACE);
+ request->tablet_schema.columns.push_back(sequence_col);
+}
+
+static TDescriptorTable create_descriptor_tablet_with_sequence_col() {
+ TDescriptorTableBuilder dtb;
+ TTupleDescriptorBuilder tuple_builder;
+
+ tuple_builder.add_slot(
+ TSlotDescriptorBuilder().type(TYPE_TINYINT).column_name("k1").column_pos(0).build());
+ tuple_builder.add_slot(
+ TSlotDescriptorBuilder().type(TYPE_SMALLINT).column_name("k2").column_pos(1).build());
+ tuple_builder.add_slot(TSlotDescriptorBuilder()
+ .type(TYPE_DATETIME)
+ .column_name("v1")
+ .column_pos(2)
+ .nullable(false)
+ .build());
+ tuple_builder.add_slot(TSlotDescriptorBuilder()
+ .type(TYPE_DATEV2)
+ .column_name("v2")
+ .column_pos(3)
+ .nullable(false)
+ .build());
+ tuple_builder.add_slot(TSlotDescriptorBuilder()
+ .type(TYPE_INT)
+ .column_name(SEQUENCE_COL)
+ .column_pos(4)
+ .nullable(false)
+ .build());
+ tuple_builder.build(&dtb);
+
+ return dtb.desc_tbl();
+}
+
+static void generate_data(vectorized::Block* block, int8_t k1, int16_t k2, int32_t seq) {
+ auto columns = block->mutate_columns();
+ int8_t c1 = k1;
+ columns[0]->insert_data((const char*)&c1, sizeof(c1));
+
+ int16_t c2 = k2;
+ columns[1]->insert_data((const char*)&c2, sizeof(c2));
+
+ vectorized::VecDateTimeValue c3;
+ c3.from_date_str("2020-07-16 19:39:43", 19);
+ int64_t c3_int = c3.to_int64();
+ columns[2]->insert_data((const char*)&c3_int, sizeof(c3));
+
+ doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType> c4;
+ c4.set_time(2022, 6, 6, 0, 0, 0, 0);
+ uint32_t c4_int = c4.to_date_int_val();
+ columns[3]->insert_data((const char*)&c4_int, sizeof(c4));
+
+ int32_t c5 = seq;
+ columns[4]->insert_data((const char*)&c5, sizeof(c2));
+}
+
+class SegmentCacheTest : public ::testing::Test {
+public:
+ SegmentCacheTest() {}
+ ~SegmentCacheTest() {}
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```suggestion
~SegmentCacheTest() = default;
```
##########
be/test/olap/segment_cache_test.cpp:
##########
@@ -0,0 +1,347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gen_cpp/AgentService_types.h>
Review Comment:
warning: 'gen_cpp/AgentService_types.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/AgentService_types.h>
^
```
##########
be/test/olap/segment_cache_test.cpp:
##########
@@ -0,0 +1,347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gen_cpp/AgentService_types.h>
+#include <gtest/gtest-message.h>
+#include <gtest/gtest-test-part.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include <iostream>
+#include <map>
+#include <memory>
+#include <mutex>
+#include <string>
+#include <utility>
+
+#include "common/config.h"
+#include "common/object_pool.h"
+#include "exec/tablet_info.h"
+#include "gen_cpp/Descriptors_types.h"
+#include "gen_cpp/Types_types.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gtest/gtest_pred_impl.h"
+#include "io/fs/local_file_system.h"
+#include "olap/data_dir.h"
+#include "olap/delta_writer.h"
+#include "olap/iterators.h"
+#include "olap/olap_define.h"
+#include "olap/options.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/segment_v2/segment.h"
+#include "olap/schema.h"
+#include "olap/segment_loader.h"
+#include "olap/storage_engine.h"
+#include "olap/tablet.h"
+#include "olap/tablet_manager.h"
+#include "olap/task/engine_publish_version_task.h"
+#include "olap/txn_manager.h"
+#include "runtime/define_primitive_type.h"
+#include "runtime/descriptor_helper.h"
+#include "runtime/descriptors.h"
+#include "runtime/exec_env.h"
+#include "vec/columns/column.h"
+#include "vec/core/block.h"
+#include "vec/core/column_with_type_and_name.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris {
+class OlapMeta;
+
+// This is DeltaWriter unit test which used by streaming load.
+// And also it should take schema change into account after streaming load.
+
+static const uint32_t MAX_PATH_LEN = 1024;
+
+static StorageEngine* k_engine = nullptr;
+
+static void set_up() {
+ char buffer[MAX_PATH_LEN];
+ EXPECT_NE(getcwd(buffer, MAX_PATH_LEN), nullptr);
+ config::storage_root_path = std::string(buffer) + "/data_test";
+ io::global_local_filesystem()->delete_and_create_directory(config::storage_root_path);
+ std::vector<StorePath> paths;
+ paths.emplace_back(config::storage_root_path, -1);
+
+ doris::EngineOptions options;
+ options.store_paths = paths;
+ Status s = doris::StorageEngine::open(options, &k_engine);
+ EXPECT_TRUE(s.ok()) << s.to_string();
+
+ ExecEnv* exec_env = doris::ExecEnv::GetInstance();
+ exec_env->set_storage_engine(k_engine);
+ k_engine->start_bg_threads();
+}
+
+static void tear_down() {
+ if (k_engine != nullptr) {
+ k_engine->stop();
+ delete k_engine;
+ k_engine = nullptr;
+ }
+ EXPECT_EQ(system("rm -rf ./data_test"), 0);
+ io::global_local_filesystem()->delete_directory(std::string(getenv("DORIS_HOME")) + "/" +
+ UNUSED_PREFIX);
+}
+
+static void create_tablet_request_with_sequence_col(int64_t tablet_id, int32_t schema_hash,
+ TCreateTabletReq* request,
+ bool enable_mow = false) {
+ request->tablet_id = tablet_id;
+ request->partition_id = 10000;
+ request->__set_version(1);
+ request->tablet_schema.schema_hash = schema_hash;
+ request->tablet_schema.short_key_column_count = 2;
+ request->tablet_schema.keys_type = TKeysType::UNIQUE_KEYS;
+ request->tablet_schema.storage_type = TStorageType::COLUMN;
+ request->tablet_schema.__set_sequence_col_idx(4);
+ request->__set_storage_format(TStorageFormat::V2);
+ request->__set_enable_unique_key_merge_on_write(enable_mow);
+
+ TColumn k1;
+ k1.column_name = "k1";
+ k1.__set_is_key(true);
+ k1.column_type.type = TPrimitiveType::TINYINT;
+ request->tablet_schema.columns.push_back(k1);
+
+ TColumn k2;
+ k2.column_name = "k2";
+ k2.__set_is_key(true);
+ k2.column_type.type = TPrimitiveType::SMALLINT;
+ request->tablet_schema.columns.push_back(k2);
+
+ TColumn v1;
+ v1.column_name = "v1";
+ v1.__set_is_key(false);
+ v1.column_type.type = TPrimitiveType::DATETIME;
+ v1.__set_aggregation_type(TAggregationType::REPLACE);
+ request->tablet_schema.columns.push_back(v1);
+
+ TColumn v2;
+ v2.column_name = "v2";
+ v2.__set_is_key(false);
+ v2.column_type.type = TPrimitiveType::DATEV2;
+ v2.__set_aggregation_type(TAggregationType::REPLACE);
+ request->tablet_schema.columns.push_back(v2);
+
+ TColumn sequence_col;
+ sequence_col.column_name = SEQUENCE_COL;
+ sequence_col.__set_is_key(false);
+ sequence_col.column_type.type = TPrimitiveType::INT;
+ sequence_col.__set_aggregation_type(TAggregationType::REPLACE);
+ request->tablet_schema.columns.push_back(sequence_col);
+}
+
+static TDescriptorTable create_descriptor_tablet_with_sequence_col() {
+ TDescriptorTableBuilder dtb;
+ TTupleDescriptorBuilder tuple_builder;
+
+ tuple_builder.add_slot(
+ TSlotDescriptorBuilder().type(TYPE_TINYINT).column_name("k1").column_pos(0).build());
+ tuple_builder.add_slot(
+ TSlotDescriptorBuilder().type(TYPE_SMALLINT).column_name("k2").column_pos(1).build());
+ tuple_builder.add_slot(TSlotDescriptorBuilder()
+ .type(TYPE_DATETIME)
+ .column_name("v1")
+ .column_pos(2)
+ .nullable(false)
+ .build());
+ tuple_builder.add_slot(TSlotDescriptorBuilder()
+ .type(TYPE_DATEV2)
+ .column_name("v2")
+ .column_pos(3)
+ .nullable(false)
+ .build());
+ tuple_builder.add_slot(TSlotDescriptorBuilder()
+ .type(TYPE_INT)
+ .column_name(SEQUENCE_COL)
+ .column_pos(4)
+ .nullable(false)
+ .build());
+ tuple_builder.build(&dtb);
+
+ return dtb.desc_tbl();
+}
+
+static void generate_data(vectorized::Block* block, int8_t k1, int16_t k2, int32_t seq) {
+ auto columns = block->mutate_columns();
+ int8_t c1 = k1;
+ columns[0]->insert_data((const char*)&c1, sizeof(c1));
+
+ int16_t c2 = k2;
+ columns[1]->insert_data((const char*)&c2, sizeof(c2));
+
+ vectorized::VecDateTimeValue c3;
+ c3.from_date_str("2020-07-16 19:39:43", 19);
+ int64_t c3_int = c3.to_int64();
+ columns[2]->insert_data((const char*)&c3_int, sizeof(c3));
+
+ doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType> c4;
+ c4.set_time(2022, 6, 6, 0, 0, 0, 0);
+ uint32_t c4_int = c4.to_date_int_val();
+ columns[3]->insert_data((const char*)&c4_int, sizeof(c4));
+
+ int32_t c5 = seq;
+ columns[4]->insert_data((const char*)&c5, sizeof(c2));
+}
+
+class SegmentCacheTest : public ::testing::Test {
+public:
+ SegmentCacheTest() {}
Review Comment:
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
```suggestion
SegmentCacheTest() = default;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37824:
URL: https://github.com/apache/doris/pull/37824#issuecomment-2228092439
<details>
<summary>TPC-DS: <b>Total hot run time: 204292 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e4eef77f951087f35bf6184618e82a865ba093a9, data reload: false
query1 934 427 380 380
query2 6579 2749 2697 2697
query3 6923 216 203 203
query4 20340 18090 17954 17954
query5 19725 6550 6581 6550
query6 298 223 230 223
query7 4154 300 313 300
query8 404 406 414 406
query9 3146 2662 2597 2597
query10 423 296 291 291
query11 11404 10707 10767 10707
query12 132 78 75 75
query13 5606 706 724 706
query14 17762 13169 13661 13169
query15 361 248 244 244
query16 6465 295 269 269
query17 1682 1468 869 869
query18 2307 419 412 412
query19 199 151 160 151
query20 83 80 81 80
query21 189 96 95 95
query22 5270 5144 5058 5058
query23 32480 31854 31990 31854
query24 6912 6507 6494 6494
query25 517 445 437 437
query26 533 168 161 161
query27 1886 300 307 300
query28 6115 2397 2320 2320
query29 2827 2583 2612 2583
query30 250 165 167 165
query31 915 722 775 722
query32 65 64 63 63
query33 417 259 268 259
query34 846 476 482 476
query35 1153 946 951 946
query36 1303 1274 1364 1274
query37 98 62 64 62
query38 3054 2901 2993 2901
query39 1370 1327 1320 1320
query40 216 96 104 96
query41 46 44 42 42
query42 82 83 79 79
query43 764 655 650 650
query44 1138 718 723 718
query45 249 238 237 237
query46 1222 965 985 965
query47 1884 1887 1660 1660
query48 1033 726 691 691
query49 625 379 378 378
query50 868 624 608 608
query51 4824 4676 4719 4676
query52 97 81 88 81
query53 447 324 317 317
query54 2671 2464 2493 2464
query55 87 90 86 86
query56 241 220 215 215
query57 1344 1172 1088 1088
query58 220 194 206 194
query59 4066 4061 3957 3957
query60 228 200 228 200
query61 103 104 108 104
query62 921 443 533 443
query63 485 354 343 343
query64 2539 1557 1481 1481
query65 3655 3553 3561 3553
query66 842 380 380 380
query67 18201 17463 15894 15894
query68 7915 658 648 648
query69 583 380 339 339
query70 1580 1482 1523 1482
query71 394 306 320 306
query72 6630 3578 3549 3549
query73 745 319 329 319
query74 6259 5825 5850 5825
query75 4596 3706 3680 3680
query76 4462 1140 1186 1140
query77 542 253 264 253
query78 12807 11571 12053 11571
query79 8054 639 654 639
query80 1955 417 398 398
query81 520 237 234 234
query82 1504 100 97 97
query83 168 133 132 132
query84 262 70 71 70
query85 1413 346 344 344
query86 364 298 284 284
query87 3244 2991 3029 2991
query88 5174 2304 2317 2304
query89 391 289 301 289
query90 1776 220 217 217
query91 173 141 143 141
query92 64 57 55 55
query93 4878 540 604 540
query94 897 215 210 210
query95 1105 1069 1050 1050
query96 654 317 324 317
query97 6489 6454 6497 6454
query98 193 186 176 176
query99 2869 865 901 865
Total cold run time: 312885 ms
Total hot run time: 204292 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org