You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2024/01/30 08:53:26 UTC
[PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
airborne12 opened a new pull request, #30574:
URL: https://github.com/apache/doris/pull/30574
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918915122
TeamCity be ut coverage result:
Function Coverage: 36.18% (8626/23840)
Line Coverage: 28.24% (70502/249638)
Region Coverage: 27.25% (36399/133552)
Branch Coverage: 24.04% (18643/77556)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9cdc96df4ba81c79d24ddf3405da16f6f0f54020_9cdc96df4ba81c79d24ddf3405da16f6f0f54020/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918744713
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 3c24263355eb957bf20b7b04e471195c5a13d0fd with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.1 seconds inserted 10000000 Rows, about 709K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918927542
<details>
<summary>ClickBench: <b>Total hot run time: 30.56 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9cdc96df4ba81c79d24ddf3405da16f6f0f54020, data reload: false
query1 0.04 0.03 0.03
query2 0.05 0.02 0.02
query3 0.23 0.06 0.06
query4 1.67 0.09 0.10
query5 0.53 0.53 0.52
query6 1.20 0.66 0.63
query7 0.02 0.01 0.01
query8 0.03 0.02 0.02
query9 0.55 0.49 0.50
query10 0.55 0.56 0.56
query11 0.12 0.08 0.09
query12 0.11 0.09 0.10
query13 0.60 0.61 0.61
query14 0.79 0.80 0.81
query15 0.80 0.78 0.77
query16 0.40 0.38 0.37
query17 0.98 1.03 0.98
query18 0.21 0.26 0.24
query19 1.88 1.75 1.79
query20 0.02 0.01 0.01
query21 15.46 0.59 0.58
query22 2.74 2.71 2.22
query23 17.34 0.80 0.75
query24 2.57 1.18 0.22
query25 0.24 0.18 0.18
query26 0.51 0.13 0.14
query27 0.05 0.06 0.05
query28 11.86 0.87 0.83
query29 12.52 3.18 3.13
query30 0.62 0.50 0.54
query31 2.79 0.33 0.34
query32 3.37 0.48 0.50
query33 3.24 3.24 3.20
query34 15.76 4.29 4.25
query35 4.31 4.26 4.28
query36 1.11 1.06 1.06
query37 0.06 0.05 0.05
query38 0.04 0.03 0.03
query39 0.03 0.02 0.01
query40 0.16 0.13 0.13
query41 0.06 0.01 0.02
query42 0.03 0.01 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.68 s
Total hot run time: 30.56 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #30574:
URL: https://github.com/apache/doris/pull/30574#discussion_r1472633922
##########
be/src/olap/rowset/segment_v2/inverted_index_cache.cpp:
##########
@@ -301,8 +88,8 @@ int64_t InvertedIndexSearcherCache::mem_consumption() {
return _policy->cache()->mem_consumption();
}
-bool InvertedIndexSearcherCache::_lookup(const InvertedIndexSearcherCache::CacheKey& key,
- InvertedIndexCacheHandle* handle) {
+bool InvertedIndexSearcherCache::lookup(const InvertedIndexSearcherCache::CacheKey& key,
Review Comment:
warning: method 'lookup' can be made static [readability-convert-member-functions-to-static]
be/src/olap/rowset/segment_v2/inverted_index_cache.h:114:
```diff
- bool lookup(const InvertedIndexSearcherCache::CacheKey& key, InvertedIndexCacheHandle* handle);
+ static bool lookup(const InvertedIndexSearcherCache::CacheKey& key, InvertedIndexCacheHandle* handle);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918700315
TeamCity be ut coverage result:
Function Coverage: 36.18% (8625/23838)
Line Coverage: 28.24% (70490/249646)
Region Coverage: 27.24% (36385/133555)
Branch Coverage: 24.03% (18638/77548)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3c24263355eb957bf20b7b04e471195c5a13d0fd_3c24263355eb957bf20b7b04e471195c5a13d0fd/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918654026
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon merged PR #30574:
URL: https://github.com/apache/doris/pull/30574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1916352740
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918823992
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918709739
<details>
<summary>TPC-H: <b>Total hot run time: 37091 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c24263355eb957bf20b7b04e471195c5a13d0fd, data reload: false
------ Round 1 ----------------------------------
q1 17630 4749 4510 4510
q2 2636 146 147 146
q3 11141 939 933 933
q4 6101 734 811 734
q5 8695 2799 2896 2799
q6 193 123 127 123
q7 1141 743 729 729
q8 9355 2025 2010 2010
q9 7240 6342 6281 6281
q10 8074 2466 2416 2416
q11 415 216 208 208
q12 773 285 291 285
q13 18028 3327 3278 3278
q14 275 242 255 242
q15 515 500 483 483
q16 456 393 417 393
q17 944 517 500 500
q18 6745 5981 5988 5981
q19 1586 1477 1383 1383
q20 630 323 351 323
q21 6978 3051 3124 3051
q22 804 312 283 283
Total cold run time: 110355 ms
Total hot run time: 37091 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4485 4499 4393 4393
q2 348 236 235 235
q3 3008 2906 2793 2793
q4 1867 1640 1646 1640
q5 5267 5275 5232 5232
q6 192 116 115 115
q7 2130 1830 1747 1747
q8 3116 3255 3252 3252
q9 8434 8293 8287 8287
q10 5827 3545 3559 3545
q11 538 458 458 458
q12 729 560 600 560
q13 6778 3099 3099 3099
q14 286 249 254 249
q15 542 502 494 494
q16 521 462 467 462
q17 1837 1644 1736 1644
q18 8054 7764 7503 7503
q19 8858 1532 1552 1532
q20 2124 1914 1905 1905
q21 4906 4588 4543 4543
q22 522 482 473 473
Total cold run time: 70369 ms
Total hot run time: 54161 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918902954
<details>
<summary>TPC-H: <b>Total hot run time: 37121 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9cdc96df4ba81c79d24ddf3405da16f6f0f54020, data reload: false
------ Round 1 ----------------------------------
q1 17687 4483 4525 4483
q2 2045 139 128 128
q3 10608 916 916 916
q4 4679 689 707 689
q5 7671 2812 2914 2812
q6 183 118 121 118
q7 1138 716 711 711
q8 9255 2022 2031 2022
q9 7224 6394 6311 6311
q10 8125 2393 2478 2393
q11 417 208 204 204
q12 736 287 286 286
q13 18020 3318 3308 3308
q14 275 258 248 248
q15 538 496 496 496
q16 467 424 424 424
q17 930 577 565 565
q18 6804 5969 5974 5969
q19 1563 1350 1417 1350
q20 592 349 332 332
q21 6709 3066 3107 3066
q22 821 313 290 290
Total cold run time: 106487 ms
Total hot run time: 37121 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4489 4469 4482 4469
q2 321 220 228 220
q3 3001 2889 2784 2784
q4 1895 1663 1674 1663
q5 5205 5243 5255 5243
q6 190 114 117 114
q7 2177 1742 1764 1742
q8 3121 3260 3305 3260
q9 8381 8294 8286 8286
q10 5841 3498 3526 3498
q11 537 454 453 453
q12 734 559 584 559
q13 11092 3079 3095 3079
q14 270 257 247 247
q15 537 501 493 493
q16 518 480 481 480
q17 1828 1677 1690 1677
q18 8003 7786 7565 7565
q19 7734 1517 1495 1495
q20 2137 1919 1887 1887
q21 4844 4634 4754 4634
q22 516 452 442 442
Total cold run time: 73371 ms
Total hot run time: 54290 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918739106
<details>
<summary>ClickBench: <b>Total hot run time: 30.12 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3c24263355eb957bf20b7b04e471195c5a13d0fd, data reload: false
query1 0.03 0.03 0.03
query2 0.05 0.02 0.02
query3 0.23 0.06 0.06
query4 1.67 0.10 0.10
query5 0.53 0.51 0.53
query6 1.18 0.64 0.67
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.56 0.49 0.51
query10 0.55 0.54 0.55
query11 0.11 0.08 0.09
query12 0.11 0.09 0.09
query13 0.60 0.61 0.60
query14 0.76 0.80 0.80
query15 0.77 0.77 0.77
query16 0.40 0.38 0.38
query17 1.01 1.02 0.98
query18 0.22 0.28 0.22
query19 1.87 1.75 1.73
query20 0.01 0.01 0.01
query21 15.41 0.56 0.58
query22 2.19 2.62 1.32
query23 17.29 0.88 0.73
query24 2.64 0.73 0.76
query25 0.41 0.17 0.13
query26 0.46 0.14 0.13
query27 0.06 0.05 0.04
query28 12.04 0.85 0.84
query29 12.56 3.17 3.13
query30 0.65 0.54 0.53
query31 2.80 0.34 0.34
query32 3.39 0.48 0.48
query33 3.21 3.21 3.22
query34 15.80 4.27 4.29
query35 4.29 4.31 4.29
query36 1.12 1.05 1.05
query37 0.06 0.04 0.04
query38 0.03 0.03 0.03
query39 0.02 0.02 0.01
query40 0.16 0.13 0.13
query41 0.07 0.02 0.01
query42 0.02 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.43 s
Total hot run time: 30.12 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1920362277
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #30574:
URL: https://github.com/apache/doris/pull/30574#discussion_r1470805614
##########
be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:
##########
@@ -34,8 +34,8 @@
static std::unique_ptr<Query> create(InvertedIndexQueryType query_type, Args&&... args) {
switch (query_type) {
case InvertedIndexQueryType::MATCH_ANY_QUERY:
- case InvertedIndexQueryType::EQUAL_QUERY:
return std::make_unique<DisjunctionQuery>(std::forward<Args>(args)...);
+ case InvertedIndexQueryType::EQUAL_QUERY:
case InvertedIndexQueryType::MATCH_ALL_QUERY:
return std::make_unique<ConjunctionQuery>(std::forward<Args>(args)...);
Review Comment:
warning: 'ConjunctionQuery' does not refer to a value [clang-diagnostic-error]
```cpp
return std::make_unique<ConjunctionQuery>(std::forward<Args>(args)...);
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:25:** declared here
```cpp
class ConjunctionQuery;
^
```
</details>
##########
be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:
##########
@@ -34,8 +34,8 @@
static std::unique_ptr<Query> create(InvertedIndexQueryType query_type, Args&&... args) {
switch (query_type) {
case InvertedIndexQueryType::MATCH_ANY_QUERY:
- case InvertedIndexQueryType::EQUAL_QUERY:
return std::make_unique<DisjunctionQuery>(std::forward<Args>(args)...);
Review Comment:
warning: no member named 'make_unique' in namespace 'std' [clang-diagnostic-error]
```cpp
return std::make_unique<DisjunctionQuery>(std::forward<Args>(args)...);
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:
##########
@@ -691,52 +549,50 @@ Status BkdIndexReader::bkd_query(OlapReaderStatistics* stats, const std::string&
return Status::Error<ErrorCode::INVERTED_INDEX_NOT_SUPPORTED>(
"invalid query type when query bkd index");
}
- visitor->set_reader(r.get());
return Status::OK();
}
-Status BkdIndexReader::invoke_bkd_try_query(OlapReaderStatistics* stats,
- const std::string& column_name, const void* query_value,
+Status BkdIndexReader::invoke_bkd_try_query(const void* query_value,
Review Comment:
warning: method 'invoke_bkd_try_query' can be made static [readability-convert-member-functions-to-static]
be/src/olap/rowset/segment_v2/inverted_index_reader.h:263:
```diff
- Status invoke_bkd_try_query(const void* query_value, InvertedIndexQueryType query_type,
+ static Status invoke_bkd_try_query(const void* query_value, InvertedIndexQueryType query_type,
```
##########
be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:
##########
@@ -34,8 +34,8 @@
static std::unique_ptr<Query> create(InvertedIndexQueryType query_type, Args&&... args) {
switch (query_type) {
case InvertedIndexQueryType::MATCH_ANY_QUERY:
- case InvertedIndexQueryType::EQUAL_QUERY:
return std::make_unique<DisjunctionQuery>(std::forward<Args>(args)...);
+ case InvertedIndexQueryType::EQUAL_QUERY:
case InvertedIndexQueryType::MATCH_ALL_QUERY:
return std::make_unique<ConjunctionQuery>(std::forward<Args>(args)...);
Review Comment:
warning: expected ')' [clang-diagnostic-error]
```cpp
return std::make_unique<ConjunctionQuery>(std::forward<Args>(args)...);
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:39:** to match this '('
```cpp
return std::make_unique<ConjunctionQuery>(std::forward<Args>(args)...);
^
```
</details>
##########
be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:
##########
@@ -34,8 +34,8 @@
static std::unique_ptr<Query> create(InvertedIndexQueryType query_type, Args&&... args) {
switch (query_type) {
case InvertedIndexQueryType::MATCH_ANY_QUERY:
- case InvertedIndexQueryType::EQUAL_QUERY:
return std::make_unique<DisjunctionQuery>(std::forward<Args>(args)...);
+ case InvertedIndexQueryType::EQUAL_QUERY:
case InvertedIndexQueryType::MATCH_ALL_QUERY:
return std::make_unique<ConjunctionQuery>(std::forward<Args>(args)...);
Review Comment:
warning: no member named 'make_unique' in namespace 'std' [clang-diagnostic-error]
```cpp
return std::make_unique<ConjunctionQuery>(std::forward<Args>(args)...);
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index_searcher.h:
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <CLucene.h>
Review Comment:
warning: 'CLucene.h' file not found [clang-diagnostic-error]
```cpp
#include <CLucene.h>
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index_cache.cpp:
##########
@@ -301,8 +88,8 @@ int64_t InvertedIndexSearcherCache::mem_consumption() {
return _policy->cache()->mem_consumption();
}
-bool InvertedIndexSearcherCache::_lookup(const InvertedIndexSearcherCache::CacheKey& key,
- InvertedIndexCacheHandle* handle) {
+bool InvertedIndexSearcherCache::lookup(const InvertedIndexSearcherCache::CacheKey& key,
Review Comment:
warning: method 'lookup' can be made static [readability-convert-member-functions-to-static]
be/src/olap/rowset/segment_v2/inverted_index_cache.h:106:
```diff
- bool lookup(const InvertedIndexSearcherCache::CacheKey& key, InvertedIndexCacheHandle* handle);
+ static bool lookup(const InvertedIndexSearcherCache::CacheKey& key, InvertedIndexCacheHandle* handle);
```
##########
be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:
##########
@@ -746,47 +602,46 @@
return Status::OK();
}
-Status BkdIndexReader::invoke_bkd_query(OlapReaderStatistics* stats, const std::string& column_name,
- const void* query_value, InvertedIndexQueryType query_type,
+Status BkdIndexReader::invoke_bkd_query(const void* query_value, InvertedIndexQueryType query_type,
Review Comment:
warning: method 'invoke_bkd_query' can be made static [readability-convert-member-functions-to-static]
be/src/olap/rowset/segment_v2/inverted_index_reader.h:265:
```diff
- Status invoke_bkd_query(const void* query_value, InvertedIndexQueryType query_type,
+ static Status invoke_bkd_query(const void* query_value, InvertedIndexQueryType query_type,
```
##########
be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:
##########
@@ -34,8 +34,8 @@ class QueryFactory {
static std::unique_ptr<Query> create(InvertedIndexQueryType query_type, Args&&... args) {
Review Comment:
warning: no template named 'unique_ptr' in namespace 'std' [clang-diagnostic-error]
```cpp
static std::unique_ptr<Query> create(InvertedIndexQueryType query_type, Args&&... args) {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:
##########
@@ -34,8 +34,8 @@
static std::unique_ptr<Query> create(InvertedIndexQueryType query_type, Args&&... args) {
switch (query_type) {
case InvertedIndexQueryType::MATCH_ANY_QUERY:
- case InvertedIndexQueryType::EQUAL_QUERY:
return std::make_unique<DisjunctionQuery>(std::forward<Args>(args)...);
Review Comment:
warning: 'DisjunctionQuery' does not refer to a value [clang-diagnostic-error]
```cpp
return std::make_unique<DisjunctionQuery>(std::forward<Args>(args)...);
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:24:** declared here
```cpp
class DisjunctionQuery;
^
```
</details>
##########
be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:
##########
@@ -34,8 +34,8 @@
static std::unique_ptr<Query> create(InvertedIndexQueryType query_type, Args&&... args) {
switch (query_type) {
case InvertedIndexQueryType::MATCH_ANY_QUERY:
- case InvertedIndexQueryType::EQUAL_QUERY:
return std::make_unique<DisjunctionQuery>(std::forward<Args>(args)...);
Review Comment:
warning: expected ')' [clang-diagnostic-error]
```cpp
return std::make_unique<DisjunctionQuery>(std::forward<Args>(args)...);
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/inverted_index/query/query_factory.h:36:** to match this '('
```cpp
return std::make_unique<DisjunctionQuery>(std::forward<Args>(args)...);
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918729696
<details>
<summary>TPC-DS: <b>Total hot run time: 174766 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c24263355eb957bf20b7b04e471195c5a13d0fd, data reload: false
query1 931 330 321 321
query2 6563 2139 1994 1994
query3 6719 201 193 193
query4 31545 22051 22093 22051
query5 4450 342 350 342
query6 246 161 161 161
query7 4624 272 255 255
query8 239 165 172 165
query9 9020 2277 2274 2274
query10 420 204 202 202
query11 18117 15497 15393 15393
query12 122 67 65 65
query13 1617 366 375 366
query14 10392 6944 7281 6944
query15 225 173 184 173
query16 7248 263 248 248
query17 1839 499 453 453
query18 1920 251 246 246
query19 171 129 129 129
query20 69 72 73 72
query21 192 126 132 126
query22 4848 4709 4771 4709
query23 31234 30126 30241 30126
query24 11557 2770 2751 2751
query25 534 302 301 301
query26 1512 135 137 135
query27 2551 275 286 275
query28 7117 1849 1853 1849
query29 1470 606 604 604
query30 283 132 138 132
query31 920 703 713 703
query32 86 57 52 52
query33 701 206 210 206
query34 1192 449 451 449
query35 867 768 745 745
query36 1331 1188 1164 1164
query37 172 57 56 56
query38 3325 3120 3144 3120
query39 1300 1269 1244 1244
query40 330 87 78 78
query41 38 36 34 34
query42 87 83 85 83
query43 542 501 488 488
query44 1026 671 687 671
query45 191 179 177 177
query46 1032 629 635 629
query47 1534 1543 1502 1502
query48 401 321 301 301
query49 1241 274 274 274
query50 669 292 303 292
query51 5256 5092 5220 5092
query52 93 83 85 83
query53 316 261 270 261
query54 257 178 180 178
query55 77 73 78 73
query56 177 171 172 171
query57 1027 906 903 903
query58 189 146 159 146
query59 2525 2456 2413 2413
query60 201 172 175 172
query61 80 81 86 81
query62 643 348 362 348
query63 273 268 244 244
query64 6403 3574 3514 3514
query65 3264 3249 3236 3236
query66 1395 319 310 310
query67 14565 14015 14138 14015
query68 4102 497 496 496
query69 418 304 314 304
query70 1570 1517 1520 1517
query71 296 205 205 205
query72 5505 3132 2821 2821
query73 683 307 309 307
query74 6688 6154 6289 6154
query75 2967 2264 2333 2264
query76 2522 1000 955 955
query77 437 227 228 227
query78 9122 8819 8630 8630
query79 2334 490 487 487
query80 1461 315 309 309
query81 519 194 200 194
query82 689 81 78 78
query83 219 117 114 114
query84 230 73 68 68
query85 1864 334 330 330
query86 515 440 420 420
query87 3425 3232 3225 3225
query88 3690 2167 2177 2167
query89 429 362 343 343
query90 2226 182 179 179
query91 150 121 117 117
query92 53 42 41 41
query93 1791 456 417 417
query94 1436 160 155 155
query95 506 446 440 440
query96 605 315 313 313
query97 4266 4096 4105 4096
query98 202 198 180 180
query99 967 675 650 650
Total cold run time: 283388 ms
Total hot run time: 174766 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918919550
<details>
<summary>TPC-DS: <b>Total hot run time: 175513 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9cdc96df4ba81c79d24ddf3405da16f6f0f54020, data reload: false
query1 930 337 333 333
query2 6557 1960 1943 1943
query3 6692 200 198 198
query4 31727 22131 22154 22131
query5 4450 350 356 350
query6 244 167 158 158
query7 4609 266 258 258
query8 258 180 175 175
query9 9142 2300 2285 2285
query10 410 209 213 209
query11 18459 15393 15288 15288
query12 122 66 65 65
query13 1616 368 370 368
query14 10318 7291 7142 7142
query15 218 178 173 173
query16 7334 261 252 252
query17 1854 499 485 485
query18 1913 267 251 251
query19 184 130 138 130
query20 77 71 72 71
query21 192 127 128 127
query22 4865 4861 4734 4734
query23 31415 30324 30423 30324
query24 8609 2780 2763 2763
query25 517 315 315 315
query26 733 134 140 134
query27 2227 276 281 276
query28 5662 1866 1857 1857
query29 917 612 602 602
query30 277 130 135 130
query31 916 706 730 706
query32 89 54 50 50
query33 498 209 213 209
query34 807 450 469 450
query35 862 762 746 746
query36 1299 1206 1134 1134
query37 94 56 55 55
query38 3275 3143 3164 3143
query39 1305 1267 1253 1253
query40 188 83 84 83
query41 36 34 35 34
query42 81 81 83 81
query43 546 466 487 466
query44 1036 688 719 688
query45 191 178 172 172
query46 1054 627 621 621
query47 1526 1520 1497 1497
query48 396 316 299 299
query49 1121 283 283 283
query50 679 313 314 313
query51 5295 5197 5143 5143
query52 89 79 80 79
query53 319 264 260 260
query54 208 179 194 179
query55 76 82 75 75
query56 187 163 154 154
query57 992 893 891 891
query58 182 184 169 169
query59 2418 2449 2213 2213
query60 200 177 181 177
query61 83 82 82 82
query62 587 346 334 334
query63 283 251 267 251
query64 4720 3741 3660 3660
query65 3252 3255 3253 3253
query66 943 326 307 307
query67 14656 14183 14393 14183
query68 3974 510 493 493
query69 449 309 303 303
query70 1552 1482 1564 1482
query71 287 211 216 211
query72 5381 3093 2876 2876
query73 682 321 322 321
query74 6571 6210 6199 6199
query75 2898 2332 2277 2277
query76 2544 909 984 909
query77 341 229 222 222
query78 9230 8927 8617 8617
query79 3392 500 493 493
query80 2162 319 324 319
query81 508 200 197 197
query82 806 82 83 82
query83 260 115 119 115
query84 282 74 76 74
query85 2023 327 321 321
query86 518 371 372 371
query87 3457 3291 3220 3220
query88 4255 2177 2178 2177
query89 424 371 347 347
query90 1934 186 184 184
query91 141 115 114 114
query92 58 45 43 43
query93 5070 435 423 423
query94 1247 159 157 157
query95 499 458 449 449
query96 645 310 310 310
query97 4251 4113 4145 4113
query98 196 197 193 193
query99 1164 681 689 681
Total cold run time: 280218 ms
Total hot run time: 175513 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refact](inverted index) refact inverted index cache to decouple with reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30574:
URL: https://github.com/apache/doris/pull/30574#issuecomment-1918932274
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 9cdc96df4ba81c79d24ddf3405da16f6f0f54020 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.3 seconds inserted 10000000 Rows, about 751K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org