You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2024/09/12 11:27:39 UTC
[PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
airborne12 opened a new pull request, #40758:
URL: https://github.com/apache/doris/pull/40758
## Proposed changes
The analyzer logic is currently distributed across the read, write, and no-index match phases of the inverted index. In this PR, we introduce an inverted index analyzer to unify these processes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40758:
URL: https://github.com/apache/doris/pull/40758#issuecomment-2346240450
<details>
<summary>TPC-H: <b>Total hot run time: 43082 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c314c82fdf311a4d7d31eb6616d5f0aa14d081e0, data reload: false
------ Round 1 ----------------------------------
q1 13243 7627 7224 7224
q2 1089 186 185 185
q3 4196 1311 1355 1311
q4 3570 1063 1034 1034
q5 3310 3173 3171 3171
q6 241 149 149 149
q7 1043 626 633 626
q8 3434 2012 2040 2012
q9 6385 6356 6311 6311
q10 3892 2590 2531 2531
q11 419 258 259 258
q12 400 219 227 219
q13 7760 3022 3051 3022
q14 289 254 261 254
q15 585 550 527 527
q16 522 424 435 424
q17 977 958 949 949
q18 7737 6862 6765 6765
q19 1295 1254 1239 1239
q20 597 338 328 328
q21 3905 3571 3554 3554
q22 1086 989 1036 989
Total cold run time: 65975 ms
Total hot run time: 43082 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7182 7153 7135 7135
q2 337 236 229 229
q3 2970 2943 2934 2934
q4 1986 1983 1990 1983
q5 5689 5461 5455 5455
q6 233 143 144 143
q7 2088 1692 1702 1692
q8 3260 3342 3350 3342
q9 8497 8475 8435 8435
q10 3412 3488 3476 3476
q11 590 495 470 470
q12 806 560 575 560
q13 4096 3038 3008 3008
q14 313 264 267 264
q15 568 528 517 517
q16 488 443 453 443
q17 1780 1751 1719 1719
q18 8107 7756 7628 7628
q19 1728 1713 1710 1710
q20 2022 1793 1805 1793
q21 5682 5486 5410 5410
q22 1108 1017 988 988
Total cold run time: 62942 ms
Total hot run time: 59334 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40758:
URL: https://github.com/apache/doris/pull/40758#issuecomment-2346267594
<details>
<summary>TPC-DS: <b>Total hot run time: 195784 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c314c82fdf311a4d7d31eb6616d5f0aa14d081e0, data reload: false
query1 917 396 395 395
query2 6495 1749 1819 1749
query3 6668 212 220 212
query4 26949 23861 23856 23856
query5 4915 554 548 548
query6 255 172 163 163
query7 4590 307 300 300
query8 283 213 224 213
query9 8578 2583 2584 2583
query10 470 301 292 292
query11 16017 15649 15509 15509
query12 161 101 100 100
query13 1685 408 393 393
query14 11679 6993 7223 6993
query15 224 174 179 174
query16 7625 475 465 465
query17 1552 598 577 577
query18 2037 298 293 293
query19 203 149 154 149
query20 133 112 113 112
query21 213 106 107 106
query22 4550 4416 4156 4156
query23 34606 33981 33931 33931
query24 9619 3169 3172 3169
query25 679 419 427 419
query26 973 164 161 161
query27 2172 289 291 289
query28 6364 2102 2076 2076
query29 963 432 445 432
query30 300 162 166 162
query31 1014 782 819 782
query32 107 59 62 59
query33 721 319 316 316
query34 929 496 482 482
query35 881 776 718 718
query36 1081 927 915 915
query37 150 88 84 84
query38 4013 3972 3862 3862
query39 1494 1394 1413 1394
query40 215 123 124 123
query41 51 49 50 49
query42 119 98 101 98
query43 490 451 461 451
query44 1312 820 791 791
query45 207 176 175 175
query46 1119 877 858 858
query47 1891 1778 1780 1778
query48 382 296 305 296
query49 1131 495 465 465
query50 955 455 448 448
query51 7156 7123 6850 6850
query52 103 91 91 91
query53 273 195 191 191
query54 744 481 483 481
query55 84 77 78 77
query56 307 281 284 281
query57 1218 1078 1072 1072
query58 250 258 260 258
query59 2781 2453 2734 2453
query60 327 303 291 291
query61 123 238 105 105
query62 925 673 665 665
query63 225 194 187 187
query64 4203 673 663 663
query65 3276 3204 3202 3202
query66 1019 295 301 295
query67 16065 15480 15492 15480
query68 3218 900 883 883
query69 451 332 342 332
query70 1151 1155 1100 1100
query71 364 355 360 355
query72 6026 3426 3359 3359
query73 613 607 597 597
query74 9315 9171 9029 9029
query75 3187 2986 3038 2986
query76 1944 912 907 907
query77 474 425 418 418
query78 9412 9281 9271 9271
query79 939 909 917 909
query80 880 860 848 848
query81 460 272 271 271
query82 270 266 270 266
query83 198 197 195 195
query84 238 119 109 109
query85 659 423 407 407
query86 311 333 318 318
query87 4326 4312 4457 4312
query88 4275 4229 4206 4206
query89 388 382 378 378
query90 1397 346 330 330
query91 130 129 124 124
query92 78 78 80 78
query93 1074 1076 1059 1059
query94 623 375 407 375
query95 485 443 434 434
query96 492 488 488 488
query97 3134 3175 3147 3147
query98 230 237 230 230
query99 1564 1306 1300 1300
Total cold run time: 277522 ms
Total hot run time: 195784 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40758:
URL: https://github.com/apache/doris/pull/40758#issuecomment-2346200351
TeamCity be ut coverage result:
Function Coverage: 36.91% (9464/25642)
Line Coverage: 28.27% (77843/275314)
Region Coverage: 27.68% (40193/145218)
Branch Coverage: 24.29% (20423/84090)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a0b94ba0e693a405368dab9549e69c7f6874f375_a0b94ba0e693a405368dab9549e69c7f6874f375/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40758:
URL: https://github.com/apache/doris/pull/40758#issuecomment-2346247236
TeamCity be ut coverage result:
Function Coverage: 36.91% (9464/25642)
Line Coverage: 28.26% (77810/275314)
Region Coverage: 27.67% (40186/145217)
Branch Coverage: 24.28% (20414/84090)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c314c82fdf311a4d7d31eb6616d5f0aa14d081e0_c314c82fdf311a4d7d31eb6616d5f0aa14d081e0/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40758:
URL: https://github.com/apache/doris/pull/40758#issuecomment-2346281431
<details>
<summary>ClickBench: <b>Total hot run time: 31.44 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c314c82fdf311a4d7d31eb6616d5f0aa14d081e0, data reload: false
query1 0.05 0.04 0.04
query2 0.07 0.04 0.04
query3 0.22 0.05 0.06
query4 1.69 0.06 0.07
query5 0.50 0.50 0.50
query6 1.14 0.73 0.73
query7 0.02 0.01 0.01
query8 0.05 0.04 0.05
query9 0.57 0.52 0.51
query10 0.57 0.60 0.56
query11 0.16 0.12 0.12
query12 0.15 0.13 0.13
query13 0.63 0.61 0.61
query14 1.45 1.47 1.44
query15 0.91 0.86 0.90
query16 0.37 0.37 0.37
query17 1.07 1.03 1.09
query18 0.23 0.20 0.20
query19 1.98 1.80 1.77
query20 0.01 0.01 0.01
query21 15.40 0.69 0.68
query22 4.07 7.83 1.59
query23 17.88 1.35 1.31
query24 2.32 0.22 0.22
query25 0.18 0.08 0.08
query26 0.28 0.18 0.17
query27 0.08 0.07 0.07
query28 13.17 1.13 1.09
query29 12.53 3.40 3.37
query30 0.24 0.06 0.06
query31 2.86 0.43 0.41
query32 3.25 0.49 0.49
query33 3.04 3.05 3.04
query34 15.43 4.34 4.34
query35 4.34 4.34 4.34
query36 0.69 0.49 0.48
query37 0.19 0.16 0.16
query38 0.16 0.15 0.15
query39 0.05 0.04 0.04
query40 0.16 0.13 0.14
query41 0.09 0.05 0.05
query42 0.06 0.06 0.05
query43 0.05 0.04 0.04
Total cold run time: 108.36 s
Total hot run time: 31.44 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40758:
URL: https://github.com/apache/doris/pull/40758#issuecomment-2347856866
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40758:
URL: https://github.com/apache/doris/pull/40758#issuecomment-2346033642
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #40758:
URL: https://github.com/apache/doris/pull/40758#discussion_r1756725355
##########
be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:
##########
@@ -145,34 +146,15 @@ class InvertedIndexColumnWriterImpl : public InvertedIndexColumnWriter {
return open_index_directory();
}
- std::unique_ptr<lucene::analysis::Analyzer> create_chinese_analyzer() {
- auto chinese_analyzer = std::make_unique<lucene::analysis::LanguageBasedAnalyzer>();
- chinese_analyzer->setLanguage(L"chinese");
- chinese_analyzer->initDict(config::inverted_index_dict_path);
-
- auto mode = get_parser_mode_string_from_properties(_index_meta->properties());
- if (mode == INVERTED_INDEX_PARSER_FINE_GRANULARITY) {
- chinese_analyzer->setMode(lucene::analysis::AnalyzerMode::All);
- } else {
- chinese_analyzer->setMode(lucene::analysis::AnalyzerMode::Default);
- }
-
- return chinese_analyzer;
- }
-
- Status create_char_string_reader(std::unique_ptr<lucene::util::Reader>& string_reader) {
- CharFilterMap char_filter_map =
- get_parser_char_filter_map_from_properties(_index_meta->properties());
- if (!char_filter_map.empty()) {
- string_reader = std::unique_ptr<lucene::util::Reader>(CharFilterFactory::create(
- char_filter_map[INVERTED_INDEX_PARSER_CHAR_FILTER_TYPE],
- new lucene::util::SStringReader<char>(),
- char_filter_map[INVERTED_INDEX_PARSER_CHAR_FILTER_PATTERN],
- char_filter_map[INVERTED_INDEX_PARSER_CHAR_FILTER_REPLACEMENT]));
- } else {
- string_reader = std::make_unique<lucene::util::SStringReader<char>>();
+ Status create_char_string_reader(std::unique_ptr<lucene::util::Reader>& string_reader,
Review Comment:
std::unique_ptr<lucene::util::Reader>* string_reader
##########
be/src/olap/rowset/segment_v2/inverted_index/analyzer/analyzer.h:
##########
@@ -0,0 +1,48 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <memory>
+
+#include "olap/inverted_index_parser.h"
+#include "olap/rowset/segment_v2/inverted_index_query_type.h"
+
+namespace lucene {
+namespace util {
+class Reader;
+}
+namespace analysis {
+class Analyzer;
+}
+} // namespace lucene
+
+namespace doris::segment_v2::inverted_index {
+class InvertedIndexAnalyzer {
+public:
+ static std::unique_ptr<lucene::util::Reader> create_reader(CharFilterMap& char_filter_map);
+
+ static std::unique_ptr<lucene::analysis::Analyzer> create_analyzer(
+ const InvertedIndexCtx* inverted_index_ctx);
+
+ static void get_analyse_result(std::vector<std::string>& analyse_result,
Review Comment:
std::vector<std::string>* analyse_result
##########
be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:
##########
@@ -207,54 +189,30 @@ class InvertedIndexColumnWriterImpl : public InvertedIndexColumnWriter {
return Status::OK();
}
- Status create_analyzer(std::unique_ptr<lucene::analysis::Analyzer>& analyzer) {
+ Status create_analyzer(std::unique_ptr<lucene::analysis::Analyzer>& analyzer,
Review Comment:
std::unique_ptr<lucene::analysis::Analyzer>* analyzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #40758:
URL: https://github.com/apache/doris/pull/40758#issuecomment-2346199127
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 merged PR #40758:
URL: https://github.com/apache/doris/pull/40758
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Refactor](inverted index) add analyzer for inverted index to unify analysis process [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #40758:
URL: https://github.com/apache/doris/pull/40758#issuecomment-2346033877
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org