You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hust-hhb (via GitHub)" <gi...@apache.org> on 2024/02/19 08:43:06 UTC
[PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
hust-hhb opened a new pull request, #31112:
URL: https://github.com/apache/doris/pull/31112
## Proposed changes
Bound checking shoud be split into tow part:
1. check column id count recorded in wal file is equal with wal block column size
2. check the output block wal reader want to construct has the same column size with _tuple_descriptor
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #31112:
URL: https://github.com/apache/doris/pull/31112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1954797729
<details>
<summary>TPC-DS: <b>Total hot run time: 186091 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cc471ab2e582acaa55a4cf74819afde5b6501a4e, data reload: false
query1 925 355 338 338
query2 6507 1756 1871 1756
query3 6704 209 205 205
query4 23213 21284 21156 21156
query5 4348 464 372 372
query6 261 177 179 177
query7 4606 301 309 301
query8 248 191 191 191
query9 8412 2794 2810 2794
query10 413 214 240 214
query11 15000 14712 14447 14447
query12 139 85 81 81
query13 1666 424 431 424
query14 9038 7791 7649 7649
query15 217 185 194 185
query16 7119 253 253 253
query17 1411 569 543 543
query18 2101 264 263 263
query19 185 142 148 142
query20 87 81 83 81
query21 189 126 124 124
query22 4759 4698 4661 4661
query23 32602 31578 31652 31578
query24 12660 3419 3422 3419
query25 644 365 365 365
query26 1864 150 158 150
query27 3057 322 314 314
query28 6882 1863 1854 1854
query29 1255 609 607 607
query30 278 139 149 139
query31 943 754 750 750
query32 102 62 54 54
query33 724 238 243 238
query34 1037 481 499 481
query35 947 836 817 817
query36 988 889 929 889
query37 174 60 61 60
query38 3321 3191 3215 3191
query39 1399 1332 1329 1329
query40 290 107 105 105
query41 36 35 36 35
query42 106 101 102 101
query43 496 448 445 445
query44 1099 697 710 697
query45 202 183 177 177
query46 1042 769 751 751
query47 1659 1533 1594 1533
query48 412 343 340 340
query49 1224 304 302 302
query50 783 372 371 371
query51 5354 5228 5142 5142
query52 106 90 95 90
query53 391 297 306 297
query54 295 216 217 216
query55 87 77 83 77
query56 220 198 218 198
query57 1083 949 935 935
query58 224 197 199 197
query59 2358 2177 2293 2177
query60 234 218 231 218
query61 84 83 89 83
query62 600 390 375 375
query63 333 288 280 280
query64 6562 3086 3130 3086
query65 3266 3272 3267 3267
query66 1376 320 319 319
query67 14456 14550 14362 14362
query68 4998 584 564 564
query69 502 359 351 351
query70 1244 1259 1208 1208
query71 355 252 248 248
query72 6319 2799 2654 2654
query73 698 309 310 309
query74 7039 6467 6478 6467
query75 3200 2553 2552 2552
query76 2991 1125 1214 1125
query77 351 237 233 233
query78 9475 8848 8787 8787
query79 961 500 493 493
query80 524 357 339 339
query81 437 206 208 206
query82 161 87 81 81
query83 140 123 128 123
query84 227 76 77 76
query85 1031 350 340 340
query86 290 303 308 303
query87 3432 3302 3301 3301
query88 2668 2260 2266 2260
query89 424 353 359 353
query90 1817 164 163 163
query91 151 128 130 128
query92 54 48 51 48
query93 966 499 505 499
query94 1102 177 187 177
query95 8800 8536 8570 8536
query96 578 265 268 265
query97 4410 4249 4296 4249
query98 222 204 195 195
query99 1060 690 733 690
Total cold run time: 277430 ms
Total hot run time: 186091 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1954677110
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1958478338
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1954805845
<details>
<summary>ClickBench: <b>Total hot run time: 31.75 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cc471ab2e582acaa55a4cf74819afde5b6501a4e, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.03
query3 0.23 0.07 0.07
query4 1.65 0.09 0.09
query5 0.49 0.48 0.48
query6 1.39 0.60 0.61
query7 0.02 0.01 0.02
query8 0.03 0.03 0.03
query9 0.54 0.45 0.46
query10 0.48 0.48 0.49
query11 0.12 0.09 0.10
query12 0.11 0.10 0.10
query13 0.60 0.58 0.58
query14 0.77 0.78 0.80
query15 0.82 0.80 0.81
query16 0.32 0.31 0.34
query17 0.90 0.87 0.88
query18 0.19 0.15 0.17
query19 1.81 1.71 1.66
query20 0.01 0.01 0.01
query21 15.42 0.64 0.60
query22 2.95 3.59 3.25
query23 17.34 1.18 1.01
query24 2.23 0.40 0.13
query25 0.48 0.06 0.06
query26 0.15 0.15 0.14
query27 0.06 0.05 0.06
query28 12.30 0.86 0.84
query29 12.54 3.34 3.36
query30 0.56 0.52 0.54
query31 2.78 0.35 0.36
query32 3.34 0.48 0.47
query33 3.15 3.15 3.17
query34 15.36 4.52 4.52
query35 4.51 4.53 4.47
query36 1.07 0.96 0.95
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.03 0.02 0.02
query40 0.18 0.14 0.14
query41 0.08 0.01 0.01
query42 0.02 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 105.27 s
Total hot run time: 31.75 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1951956636
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1954779176
<details>
<summary>TPC-H: <b>Total hot run time: 41962 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cc471ab2e582acaa55a4cf74819afde5b6501a4e, data reload: false
------ Round 1 ----------------------------------
q1 18725 5296 5200 5200
q2 2352 144 139 139
q3 11125 1069 1040 1040
q4 4761 1012 1016 1012
q5 8232 3323 3292 3292
q6 193 135 137 135
q7 1338 785 760 760
q8 9354 2076 2099 2076
q9 7578 6763 6692 6692
q10 8332 2678 2632 2632
q11 407 229 224 224
q12 761 335 333 333
q13 18012 3669 3667 3667
q14 292 260 259 259
q15 590 547 506 506
q16 486 416 431 416
q17 934 871 883 871
q18 7349 6696 6625 6625
q19 1557 1495 1503 1495
q20 558 358 346 346
q21 5836 3896 4070 3896
q22 772 368 346 346
Total cold run time: 109544 ms
Total hot run time: 41962 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4873 4851 4852 4851
q2 292 183 179 179
q3 3603 3580 3578 3578
q4 2517 2520 2505 2505
q5 5745 5759 5829 5759
q6 223 123 130 123
q7 2239 1688 1701 1688
q8 3019 3086 3115 3086
q9 8714 8758 8682 8682
q10 5576 4244 4252 4244
q11 514 393 386 386
q12 772 553 552 552
q13 4166 3421 3389 3389
q14 258 233 231 231
q15 610 510 505 505
q16 487 425 434 425
q17 1671 1596 1601 1596
q18 8337 7571 7672 7571
q19 1632 1636 1640 1636
q20 2111 1862 1832 1832
q21 6528 6156 6114 6114
q22 572 526 517 517
Total cold run time: 64459 ms
Total hot run time: 59449 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1954811779
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit cc471ab2e582acaa55a4cf74819afde5b6501a4e with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1951971528
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1951957901
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1951993951
TeamCity be ut coverage result:
Function Coverage: 35.77% (8555/23916)
Line Coverage: 27.71% (69363/250306)
Region Coverage: 26.83% (35995/134143)
Branch Coverage: 23.64% (18412/77878)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f65fe0833bf442623a34a4d6a9034f9844ab9bac_f65fe0833bf442623a34a4d6a9034f9844ab9bac/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31112:
URL: https://github.com/apache/doris/pull/31112#discussion_r1496161120
##########
be/test/vec/exec/vwal_scanner_test.cpp:
##########
@@ -71,12 +79,19 @@
ScannerCounter _counter;
std::vector<TExpr> _pre_filter;
TPlanNode _tnode;
- TUniqueId unique_id;
- TQueryOptions query_options;
- TQueryGlobals query_globals;
+ TUniqueId _unique_id;
+ TQueryOptions _query_options;
+ TQueryGlobals _query_globals;
+ QueryContext* _query_ctx = nullptr;
+ std::shared_ptr<NewFileScanNode> _scan_node = nullptr;
+ std::vector<TFileRangeDesc> _ranges;
+ TFileRangeDesc _range_desc;
+ TFileScanRange _scan_range;
+ std::unique_ptr<ShardedKVCache> _kv_cache = nullptr;
+ std::unique_ptr<TMasterInfo> _master_info = nullptr;
};
-void VWalScannerTest::init_desc_table() {
+void VWalScannerTest::_init_desc_table() {
Review Comment:
warning: function '_init_desc_table' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void VWalScannerTest::_init_desc_table() {
^
```
<details>
<summary>Additional context</summary>
**be/test/vec/exec/vwal_scanner_test.cpp:93:** 117 lines including whitespace and comments (threshold 80)
```cpp
void VWalScannerTest::_init_desc_table() {
^
```
</details>
##########
be/test/vec/exec/vwal_scanner_test.cpp:
##########
@@ -71,12 +79,19 @@ class VWalScannerTest : public testing::Test {
ScannerCounter _counter;
std::vector<TExpr> _pre_filter;
TPlanNode _tnode;
- TUniqueId unique_id;
- TQueryOptions query_options;
- TQueryGlobals query_globals;
+ TUniqueId _unique_id;
+ TQueryOptions _query_options;
+ TQueryGlobals _query_globals;
+ QueryContext* _query_ctx = nullptr;
+ std::shared_ptr<NewFileScanNode> _scan_node = nullptr;
+ std::vector<TFileRangeDesc> _ranges;
+ TFileRangeDesc _range_desc;
+ TFileScanRange _scan_range;
+ std::unique_ptr<ShardedKVCache> _kv_cache = nullptr;
+ std::unique_ptr<TMasterInfo> _master_info = nullptr;
};
-void VWalScannerTest::init_desc_table() {
+void VWalScannerTest::_init_desc_table() {
Review Comment:
warning: method '_init_desc_table' can be made static [readability-convert-member-functions-to-static]
be/test/vec/exec/vwal_scanner_test.cpp:61:
```diff
- void _init_desc_table();
+ static void _init_desc_table();
```
##########
be/test/vec/exec/vwal_scanner_test.cpp:
##########
@@ -212,64 +227,87 @@
_tnode.file_scan_node.tuple_id = 0;
_tnode.__isset.file_scan_node = true;
+ _scan_node = std::make_shared<NewFileScanNode>(&_obj_pool, _tnode, *_desc_tbl);
+ _scan_node->_output_tuple_desc = _runtime_state.desc_tbl().get_tuple_descriptor(_dst_tuple_id);
+ WARN_IF_ERROR(_scan_node->init(_tnode, &_runtime_state), "fail to init scan_node");
+ WARN_IF_ERROR(_scan_node->prepare(&_runtime_state), "fail to prepare scan_node");
+
+ _range_desc.start_offset = 0;
+ _range_desc.size = 1000;
+ _ranges.push_back(_range_desc);
+ _scan_range.ranges = _ranges;
+ _scan_range.__isset.params = true;
+ _scan_range.params.format_type = TFileFormatType::FORMAT_WAL;
+ _kv_cache.reset(new ShardedKVCache(48));
+
+ _runtime_state._wal_id = _txn_id;
+
+ _master_info.reset(new TMasterInfo());
_env = ExecEnv::GetInstance();
- _env->_wal_manager = WalManager::create_shared(_env, wal_dir);
+ _env->_master_info = _master_info.get();
+ _env->_master_info->network_address.hostname = "host name";
+ _env->_master_info->network_address.port = _backend_id;
+ _env->_master_info->backend_id = 1001;
+ _env->_wal_manager = WalManager::create_shared(_env, _wal_dir);
std::string base_path;
auto st = _env->_wal_manager->_init_wal_dirs_info();
- st = _env->_wal_manager->create_wal_path(db_id, tb_id, txn_id, label, base_path);
+ st = _env->_wal_manager->create_wal_path(_db_id, _tb_id, _txn_id, _label, base_path);
+ std::string src = "./be/test/exec/test_data/wal_scanner/wal";
+ std::string dst = _wal_dir + "/" + std::to_string(_db_id) + "/" + std::to_string(_tb_id) + "/" +
+ std::to_string(_version) + "_" + std::to_string(_backend_id) + "_" +
+ std::to_string(_txn_id) + "_" + _label;
+ std::filesystem::copy(src, dst);
}
-TEST_F(VWalScannerTest, normal) {
- std::vector<size_t> index_vector;
- index_vector.emplace_back(0);
- index_vector.emplace_back(1);
- index_vector.emplace_back(2);
- // config::group_commit_replay_wal_dir = wal_dir;
- NewFileScanNode scan_node(&_obj_pool, _tnode, *_desc_tbl);
- scan_node._output_tuple_desc = _runtime_state.desc_tbl().get_tuple_descriptor(_dst_tuple_id);
- static_cast<void>(scan_node.init(_tnode, &_runtime_state));
- auto status = scan_node.prepare(&_runtime_state);
- EXPECT_TRUE(status.ok());
-
- std::vector<TFileRangeDesc> ranges;
- TFileRangeDesc range_desc;
- {
- range_desc.start_offset = 0;
- range_desc.size = 1000;
- }
- ranges.push_back(range_desc);
- TFileScanRange scan_range;
- scan_range.ranges = ranges;
- scan_range.__isset.params = true;
- scan_range.params.format_type = TFileFormatType::FORMAT_WAL;
- std::unique_ptr<ShardedKVCache> _kv_cache;
- _kv_cache.reset(new ShardedKVCache(48));
- _runtime_state._wal_id = txn_id;
- VFileScanner scanner(&_runtime_state, &scan_node, -1, scan_range, _profile, _kv_cache.get());
- scanner._is_load = false;
+void VWalScannerTest::generate_scanner(std::shared_ptr<VFileScanner>& scanner) {
Review Comment:
warning: method 'generate_scanner' can be made static [readability-convert-member-functions-to-static]
be/test/vec/exec/vwal_scanner_test.cpp:48:
```diff
- void generate_scanner(std::shared_ptr<VFileScanner>& scanner);
+ static void generate_scanner(std::shared_ptr<VFileScanner>& scanner);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31112:
URL: https://github.com/apache/doris/pull/31112#discussion_r1496213132
##########
be/test/vec/exec/vwal_scanner_test.cpp:
##########
@@ -210,73 +225,87 @@
_tnode.file_scan_node.tuple_id = 0;
_tnode.__isset.file_scan_node = true;
+ _scan_node = std::make_shared<NewFileScanNode>(&_obj_pool, _tnode, *_desc_tbl);
+ _scan_node->_output_tuple_desc = _runtime_state.desc_tbl().get_tuple_descriptor(_dst_tuple_id);
+ WARN_IF_ERROR(_scan_node->init(_tnode, &_runtime_state), "fail to init scan_node");
+ WARN_IF_ERROR(_scan_node->prepare(&_runtime_state), "fail to prepare scan_node");
+
+ _range_desc.start_offset = 0;
+ _range_desc.size = 1000;
+ _ranges.push_back(_range_desc);
+ _scan_range.ranges = _ranges;
+ _scan_range.__isset.params = true;
+ _scan_range.params.format_type = TFileFormatType::FORMAT_WAL;
+ _kv_cache.reset(new ShardedKVCache(48));
+
+ _runtime_state._wal_id = _txn_id;
+
+ _master_info.reset(new TMasterInfo());
_env = ExecEnv::GetInstance();
- _env->_master_info = new TMasterInfo();
+ _env->_master_info = _master_info.get();
_env->_master_info->network_address.hostname = "host name";
- _env->_master_info->network_address.port = backend_id;
+ _env->_master_info->network_address.port = _backend_id;
_env->_master_info->backend_id = 1001;
- _env->_wal_manager = WalManager::create_shared(_env, wal_dir);
+ _env->_wal_manager = WalManager::create_shared(_env, _wal_dir);
std::string base_path;
auto st = _env->_wal_manager->_init_wal_dirs_info();
- st = _env->_wal_manager->create_wal_path(db_id, tb_id, txn_id, label, base_path);
+ st = _env->_wal_manager->create_wal_path(_db_id, _tb_id, _txn_id, _label, base_path);
std::string src = "./be/test/exec/test_data/wal_scanner/wal";
- std::string dst = wal_dir + "/" + std::to_string(db_id) + "/" + std::to_string(tb_id) + "/" +
- std::to_string(version) + "_" + std::to_string(backend_id) + "_" +
- std::to_string(txn_id) + "_" + label;
+ std::string dst = _wal_dir + "/" + std::to_string(_db_id) + "/" + std::to_string(_tb_id) + "/" +
+ std::to_string(_version) + "_" + std::to_string(_backend_id) + "_" +
+ std::to_string(_txn_id) + "_" + _label;
std::filesystem::copy(src, dst);
}
-TEST_F(VWalScannerTest, normal) {
- std::vector<size_t> index_vector;
- index_vector.emplace_back(0);
- index_vector.emplace_back(1);
- index_vector.emplace_back(2);
- // config::group_commit_replay_wal_dir = wal_dir;
- NewFileScanNode scan_node(&_obj_pool, _tnode, *_desc_tbl);
- scan_node._output_tuple_desc = _runtime_state.desc_tbl().get_tuple_descriptor(_dst_tuple_id);
- static_cast<void>(scan_node.init(_tnode, &_runtime_state));
- auto status = scan_node.prepare(&_runtime_state);
- EXPECT_TRUE(status.ok());
-
- std::vector<TFileRangeDesc> ranges;
- TFileRangeDesc range_desc;
- {
- range_desc.start_offset = 0;
- range_desc.size = 1000;
- }
- ranges.push_back(range_desc);
- TFileScanRange scan_range;
- scan_range.ranges = ranges;
- scan_range.__isset.params = true;
- scan_range.params.format_type = TFileFormatType::FORMAT_WAL;
- std::unique_ptr<ShardedKVCache> _kv_cache;
- _kv_cache.reset(new ShardedKVCache(48));
- _runtime_state._wal_id = txn_id;
- VFileScanner scanner(&_runtime_state, &scan_node, -1, scan_range, _profile, _kv_cache.get());
- scanner._is_load = false;
+void VWalScannerTest::generate_scanner(std::shared_ptr<VFileScanner>& scanner) {
Review Comment:
warning: method 'generate_scanner' can be made static [readability-convert-member-functions-to-static]
be/test/vec/exec/vwal_scanner_test.cpp:47:
```diff
- void generate_scanner(std::shared_ptr<VFileScanner>& scanner);
+ static void generate_scanner(std::shared_ptr<VFileScanner>& scanner);
```
##########
be/test/vec/exec/vwal_scanner_test.cpp:
##########
@@ -73,12 +78,18 @@
ScannerCounter _counter;
std::vector<TExpr> _pre_filter;
TPlanNode _tnode;
- TUniqueId unique_id;
- TQueryOptions query_options;
- TQueryGlobals query_globals;
+ TUniqueId _unique_id;
+ TQueryOptions _query_options;
+ TQueryGlobals _query_globals;
+ std::shared_ptr<NewFileScanNode> _scan_node = nullptr;
+ std::vector<TFileRangeDesc> _ranges;
+ TFileRangeDesc _range_desc;
+ TFileScanRange _scan_range;
+ std::unique_ptr<ShardedKVCache> _kv_cache = nullptr;
+ std::unique_ptr<TMasterInfo> _master_info = nullptr;
};
-void VWalScannerTest::init_desc_table() {
+void VWalScannerTest::_init_desc_table() {
Review Comment:
warning: function '_init_desc_table' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void VWalScannerTest::_init_desc_table() {
^
```
<details>
<summary>Additional context</summary>
**be/test/vec/exec/vwal_scanner_test.cpp:91:** 117 lines including whitespace and comments (threshold 80)
```cpp
void VWalScannerTest::_init_desc_table() {
^
```
</details>
##########
be/test/vec/exec/vwal_scanner_test.cpp:
##########
@@ -73,12 +78,18 @@ class VWalScannerTest : public testing::Test {
ScannerCounter _counter;
std::vector<TExpr> _pre_filter;
TPlanNode _tnode;
- TUniqueId unique_id;
- TQueryOptions query_options;
- TQueryGlobals query_globals;
+ TUniqueId _unique_id;
+ TQueryOptions _query_options;
+ TQueryGlobals _query_globals;
+ std::shared_ptr<NewFileScanNode> _scan_node = nullptr;
+ std::vector<TFileRangeDesc> _ranges;
+ TFileRangeDesc _range_desc;
+ TFileScanRange _scan_range;
+ std::unique_ptr<ShardedKVCache> _kv_cache = nullptr;
+ std::unique_ptr<TMasterInfo> _master_info = nullptr;
};
-void VWalScannerTest::init_desc_table() {
+void VWalScannerTest::_init_desc_table() {
Review Comment:
warning: method '_init_desc_table' can be made static [readability-convert-member-functions-to-static]
be/test/vec/exec/vwal_scanner_test.cpp:60:
```diff
- void _init_desc_table();
+ static void _init_desc_table();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1954712728
TeamCity be ut coverage result:
Function Coverage: 35.74% (8551/23927)
Line Coverage: 27.68% (69323/250434)
Region Coverage: 26.81% (35988/134229)
Branch Coverage: 23.61% (18398/77914)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cc471ab2e582acaa55a4cf74819afde5b6501a4e_cc471ab2e582acaa55a4cf74819afde5b6501a4e/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Fix bound checking problem when reading wal block [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31112:
URL: https://github.com/apache/doris/pull/31112#issuecomment-1955806162
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org