You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2023/11/21 11:15:44 UTC
[PR] [Improvement](inverted index) enable bkd index reader cache and refactor inverted index searcher cache [doris]
airborne12 opened a new pull request, #27367:
URL: https://github.com/apache/doris/pull/27367
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](inverted index) enable bkd index reader cache and refactor inverted index searcher cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27367:
URL: https://github.com/apache/doris/pull/27367#issuecomment-1820987279
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.35 seconds
stream load tsv: 589 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 30.4 seconds inserted 10000000 Rows, about 328K ops/s
storage size: 17100033800 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](inverted index) enable bkd index reader cache and refactor inverted index searcher cache [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #27367:
URL: https://github.com/apache/doris/pull/27367#issuecomment-1820778751
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](inverted index) enable bkd index reader cache and refactor inverted index searcher cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27367:
URL: https://github.com/apache/doris/pull/27367#discussion_r1400469397
##########
be/src/olap/rowset/segment_v2/inverted_index_cache.h:
##########
@@ -54,16 +56,44 @@ namespace lucene {
namespace search {
class IndexSearcher;
} // namespace search
+namespace util {
+namespace bkd {
Review Comment:
warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]
```suggestion
namespace util::bkd {
```
be/src/olap/rowset/segment_v2/inverted_index_cache.h:61:
```diff
- }
- } // namespace util
+ } // namespace util
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](inverted index) enable bkd index reader cache and refactor inverted index searcher cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27367:
URL: https://github.com/apache/doris/pull/27367#issuecomment-1820966733
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit cca061c27b632987726159f33b6f95a18f98192f, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4870 4665 4626 4626
q2 352 148 168 148
q3 2046 1855 1870 1855
q4 1374 1233 1263 1233
q5 3963 3920 4013 3920
q6 244 127 128 127
q7 1414 891 905 891
q8 2734 2768 2756 2756
q9 9713 9658 9507 9507
q10 3459 3507 3537 3507
q11 381 258 252 252
q12 443 295 298 295
q13 4542 3822 3767 3767
q14 332 300 281 281
q15 580 535 532 532
q16 658 575 583 575
q17 1124 982 922 922
q18 7817 7411 7410 7410
q19 1672 1665 1664 1664
q20 564 299 295 295
q21 4323 3964 3948 3948
q22 485 373 386 373
Total cold run time: 53090 ms
Total hot run time: 48884 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4612 4542 4573 4542
q2 338 214 296 214
q3 3991 3974 3994 3974
q4 2689 2701 2690 2690
q5 9731 9710 9734 9710
q6 237 121 123 121
q7 3011 2480 2472 2472
q8 4477 4461 4498 4461
q9 13244 13096 13171 13096
q10 4098 4197 4170 4170
q11 825 709 664 664
q12 976 805 813 805
q13 4258 3589 3547 3547
q14 395 335 349 335
q15 575 531 532 531
q16 739 667 685 667
q17 3873 3852 3875 3852
q18 9591 8883 8923 8883
q19 1811 1766 1765 1765
q20 2394 2093 2040 2040
q21 8819 8648 8506 8506
q22 911 815 792 792
Total cold run time: 81595 ms
Total hot run time: 77837 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](inverted index) enable bkd index reader cache and refactor inverted index searcher cache [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 closed pull request #27367: [Improvement](inverted index) enable bkd index reader cache and refactor inverted index searcher cache
URL: https://github.com/apache/doris/pull/27367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](inverted index) enable bkd index reader cache and refactor inverted index searcher cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27367:
URL: https://github.com/apache/doris/pull/27367#discussion_r1400427631
##########
be/src/olap/rowset/segment_v2/inverted_index_cache.cpp:
##########
@@ -39,19 +42,78 @@
namespace doris {
namespace segment_v2 {
-IndexSearcherPtr InvertedIndexSearcherCache::build_index_searcher(const io::FileSystemSPtr& fs,
- const std::string& index_dir,
- const std::string& file_name) {
+Status FulltextIndexSearcherBuilder::build(const io::FileSystemSPtr& fs,
+ const std::string& index_dir,
+ const std::string& file_name,
+ OptionalIndexSearcherPtr& output_searcher) {
DorisCompoundReader* directory =
new DorisCompoundReader(DorisCompoundDirectory::getDirectory(fs, index_dir.c_str()),
file_name.c_str(), config::inverted_index_read_buffer_size);
auto closeDirectory = true;
auto index_searcher =
std::make_shared<lucene::search::IndexSearcher>(directory, closeDirectory);
+ if (!index_searcher) {
+ _CLDECDELETE(directory)
+ output_searcher = std::nullopt;
+ return Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
+ "FulltextIndexSearcherBuilder build index_searcher error.");
+ }
// NOTE: need to cl_refcount-- here, so that directory will be deleted when
// index_searcher is destroyed
_CLDECDELETE(directory)
- return index_searcher;
+ output_searcher = index_searcher;
+ return Status::OK();
+}
+
+Status BKDIndexSearcherBuilder::build(const io::FileSystemSPtr& fs, const std::string& index_dir,
Review Comment:
warning: method 'build' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status BKDIndexSearcherBuilder::build(const io::FileSystemSPtr& fs, const std::string& index_dir,
```
##########
be/src/olap/rowset/segment_v2/inverted_index_cache.h:
##########
@@ -54,16 +55,44 @@ namespace lucene {
namespace search {
class IndexSearcher;
} // namespace search
+namespace util {
+namespace bkd {
Review Comment:
warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]
```suggestion
namespace util::bkd {
```
be/src/olap/rowset/segment_v2/inverted_index_cache.h:60:
```diff
- }
- } // namespace util
+ } // namespace util
```
##########
be/src/olap/rowset/segment_v2/inverted_index_cache.cpp:
##########
@@ -39,19 +42,78 @@
namespace doris {
namespace segment_v2 {
-IndexSearcherPtr InvertedIndexSearcherCache::build_index_searcher(const io::FileSystemSPtr& fs,
- const std::string& index_dir,
- const std::string& file_name) {
+Status FulltextIndexSearcherBuilder::build(const io::FileSystemSPtr& fs,
Review Comment:
warning: method 'build' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status FulltextIndexSearcherBuilder::build(const io::FileSystemSPtr& fs,
```
##########
be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:
##########
@@ -479,7 +484,7 @@ Status FullTextIndexReader::match_all_index_search(
return Status::OK();
}
-void FullTextIndexReader::check_null_bitmap(const IndexSearcherPtr& index_searcher,
+void FullTextIndexReader::check_null_bitmap(const FulltextIndexSearcherPtr& index_searcher,
Review Comment:
warning: method 'check_null_bitmap' can be made static [readability-convert-member-functions-to-static]
be/src/olap/rowset/segment_v2/inverted_index_reader.h:145:
```diff
- void check_null_bitmap(const FulltextIndexSearcherPtr& index_searcher,
+ static void check_null_bitmap(const FulltextIndexSearcherPtr& index_searcher,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org