You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zzzxl1993 (via GitHub)" <gi...@apache.org> on 2024/10/30 11:50:57 UTC
[PR] [opt](inverted index) New algorithm implementation for "match_phrase slop" [doris]
zzzxl1993 opened a new pull request, #42932:
URL: https://github.com/apache/doris/pull/42932
## Proposed changes
1. Reimplement using a higher version of Lucene's PhraseQuery
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) New algorithm implementation for "match_phrase slop" [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #42932:
URL: https://github.com/apache/doris/pull/42932#discussion_r1825659481
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
+
+ void set(int32_t i) { derived()->set_impl(i); }
+
+ void clear(int32_t i) { derived()->clear_impl(i); }
+
+ void clear(int32_t start_index, int32_t end_index) {
+ derived()->clear_impl(start_index, end_index);
Review Comment:
warning: use of undeclared identifier 'derived'; did you mean 'Derived'? [clang-diagnostic-error]
```cpp
derived()->clear_impl(start_index, end_index);
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:23:** 'Derived' declared here
```cpp
template <typename Derived>
^
```
</details>
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
+
+ void set(int32_t i) { derived()->set_impl(i); }
+
+ void clear(int32_t i) { derived()->clear_impl(i); }
+
+ void clear(int32_t start_index, int32_t end_index) {
Review Comment:
warning: unknown type name 'int32_t' [clang-diagnostic-error]
```cpp
void clear(int32_t start_index, int32_t end_index) {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
+
+ void set(int32_t i) { derived()->set_impl(i); }
+
+ void clear(int32_t i) { derived()->clear_impl(i); }
+
+ void clear(int32_t start_index, int32_t end_index) {
Review Comment:
warning: unknown type name 'int32_t' [clang-diagnostic-error]
```cpp
void clear(int32_t start_index, int32_t end_index) {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bits.h:
##########
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class Bits {
+public:
+ bool get(int32_t index) { derived()->get_impl(); }
Review Comment:
warning: unknown type name 'int32_t' [clang-diagnostic-error]
```cpp
bool get(int32_t index) { derived()->get_impl(); }
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bits.h:
##########
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class Bits {
+public:
+ bool get(int32_t index) { derived()->get_impl(); }
+
+ int32_t length() { derived()->length_impl(); }
Review Comment:
warning: unknown type name 'int32_t' [clang-diagnostic-error]
```cpp
int32_t length() { derived()->length_impl(); }
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2700453126
<details>
<summary>ClickBench: <b>Total hot run time: 31.16 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d45a8921e34cb932904b95ebdde40784225b1cd, data reload: false
query1 0.03 0.03 0.03
query2 0.07 0.03 0.03
query3 0.24 0.06 0.07
query4 1.63 0.10 0.10
query5 0.55 0.57 0.54
query6 1.20 0.72 0.72
query7 0.03 0.01 0.02
query8 0.05 0.04 0.04
query9 0.59 0.56 0.53
query10 0.57 0.58 0.58
query11 0.15 0.10 0.11
query12 0.14 0.11 0.12
query13 0.61 0.60 0.60
query14 2.69 2.82 2.69
query15 0.92 0.86 0.85
query16 0.37 0.37 0.38
query17 1.03 1.05 1.03
query18 0.21 0.20 0.19
query19 1.99 2.02 1.82
query20 0.02 0.01 0.01
query21 15.39 0.89 0.55
query22 0.76 1.15 0.69
query23 14.95 1.40 0.61
query24 7.41 1.35 1.01
query25 0.50 0.17 0.15
query26 0.56 0.16 0.15
query27 0.06 0.05 0.04
query28 9.19 0.90 0.44
query29 12.54 4.01 3.32
query30 0.24 0.08 0.06
query31 2.83 0.56 0.38
query32 3.23 0.54 0.47
query33 2.96 3.06 3.08
query34 15.72 5.18 4.50
query35 4.56 4.52 4.58
query36 0.66 0.49 0.48
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.13 0.13
query41 0.08 0.02 0.02
query42 0.03 0.03 0.02
query43 0.03 0.03 0.03
Total cold run time: 105.13 s
Total hot run time: 31.16 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2791530471
<details>
<summary>TPC-DS: <b>Total hot run time: 193384 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 49520d7e9cdd2eda186cfa5815b3db442739314f, data reload: false
query1 1397 1077 1044 1044
query2 6051 2013 1963 1963
query3 11140 4608 4616 4608
query4 55734 25375 23051 23051
query5 5567 552 474 474
query6 380 213 206 206
query7 4996 497 275 275
query8 321 234 228 228
query9 6331 2535 2571 2535
query10 410 325 259 259
query11 15088 15113 15034 15034
query12 162 112 107 107
query13 1132 529 406 406
query14 10185 6463 6420 6420
query15 184 209 189 189
query16 7027 660 465 465
query17 1083 736 575 575
query18 1572 444 306 306
query19 194 200 169 169
query20 125 130 126 126
query21 202 122 103 103
query22 4481 4565 4468 4468
query23 34191 33583 33201 33201
query24 6588 2436 2441 2436
query25 464 476 388 388
query26 652 289 153 153
query27 2175 505 339 339
query28 3277 2477 2449 2449
query29 587 560 446 446
query30 275 226 188 188
query31 884 880 798 798
query32 78 62 66 62
query33 458 385 307 307
query34 785 888 510 510
query35 847 845 791 791
query36 965 1042 912 912
query37 128 105 79 79
query38 4245 4167 4229 4167
query39 1514 1476 1446 1446
query40 210 124 106 106
query41 53 56 51 51
query42 130 109 108 108
query43 516 525 510 510
query44 1359 822 833 822
query45 195 178 167 167
query46 832 1027 652 652
query47 1843 1872 1834 1834
query48 386 417 326 326
query49 686 537 454 454
query50 671 716 403 403
query51 4318 4243 4306 4243
query52 110 117 100 100
query53 241 267 194 194
query54 571 569 519 519
query55 82 88 88 88
query56 299 306 295 295
query57 1195 1178 1129 1129
query58 277 261 262 261
query59 2794 2908 2794 2794
query60 343 325 330 325
query61 135 130 152 130
query62 750 758 678 678
query63 225 188 194 188
query64 1477 1070 710 710
query65 4413 4244 4235 4235
query66 735 403 303 303
query67 15962 15612 15445 15445
query68 4676 895 519 519
query69 500 300 272 272
query70 1222 1150 1122 1122
query71 454 329 297 297
query72 5829 4827 4972 4827
query73 758 680 355 355
query74 8943 8995 8659 8659
query75 3185 3213 2757 2757
query76 3217 1201 788 788
query77 562 373 299 299
query78 10032 9851 9211 9211
query79 2539 803 570 570
query80 722 563 448 448
query81 483 264 223 223
query82 528 135 94 94
query83 271 268 233 233
query84 294 112 85 85
query85 798 355 308 308
query86 430 332 296 296
query87 4517 4473 4292 4292
query88 3769 2227 2238 2227
query89 407 321 288 288
query90 1670 219 223 219
query91 144 144 111 111
query92 72 62 57 57
query93 2709 935 579 579
query94 723 422 282 282
query95 375 304 300 300
query96 479 574 280 280
query97 3168 3194 3157 3157
query98 230 210 208 208
query99 1328 1418 1268 1268
Total cold run time: 297717 ms
Total hot run time: 193384 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2791536470
<details>
<summary>ClickBench: <b>Total hot run time: 30.94 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 49520d7e9cdd2eda186cfa5815b3db442739314f, data reload: false
query1 0.04 0.03 0.03
query2 0.12 0.11 0.10
query3 0.25 0.20 0.20
query4 1.59 0.19 0.20
query5 0.60 0.58 0.62
query6 1.18 0.73 0.71
query7 0.03 0.02 0.02
query8 0.04 0.04 0.03
query9 0.59 0.52 0.52
query10 0.59 0.59 0.56
query11 0.16 0.11 0.10
query12 0.15 0.11 0.12
query13 0.61 0.61 0.61
query14 2.68 2.70 2.71
query15 0.92 0.86 0.86
query16 0.38 0.38 0.40
query17 1.02 1.03 1.03
query18 0.21 0.20 0.20
query19 1.94 1.89 1.81
query20 0.01 0.01 0.02
query21 15.34 0.87 0.56
query22 0.74 1.27 0.76
query23 14.74 1.41 0.60
query24 7.11 1.41 0.40
query25 0.35 0.16 0.08
query26 0.57 0.16 0.15
query27 0.05 0.05 0.05
query28 8.62 0.93 0.46
query29 12.53 3.97 3.26
query30 0.25 0.09 0.06
query31 2.82 0.60 0.39
query32 3.23 0.56 0.47
query33 3.04 3.10 3.14
query34 15.79 5.18 4.47
query35 4.57 4.57 4.59
query36 0.69 0.50 0.48
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.16 0.13 0.13
query41 0.09 0.03 0.02
query42 0.03 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 104.04 s
Total hot run time: 30.94 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2701212080
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2701390238
<details>
<summary>ClickBench: <b>Total hot run time: 30.57 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4bf31488c6c90a028a6d62a484c8bc8a1f2b5539, data reload: false
query1 0.03 0.03 0.03
query2 0.07 0.03 0.03
query3 0.24 0.07 0.07
query4 1.62 0.10 0.11
query5 0.58 0.54 0.56
query6 1.19 0.72 0.73
query7 0.03 0.02 0.01
query8 0.04 0.04 0.03
query9 0.62 0.52 0.52
query10 0.56 0.57 0.57
query11 0.16 0.11 0.11
query12 0.14 0.12 0.12
query13 0.61 0.60 0.61
query14 2.67 2.68 2.70
query15 0.92 0.86 0.84
query16 0.37 0.38 0.39
query17 1.01 1.00 1.02
query18 0.22 0.19 0.20
query19 1.93 1.87 1.96
query20 0.01 0.01 0.01
query21 15.39 0.90 0.56
query22 0.75 1.19 0.64
query23 15.00 1.38 0.62
query24 7.13 2.08 0.64
query25 0.50 0.28 0.14
query26 0.64 0.16 0.14
query27 0.06 0.06 0.05
query28 10.10 0.87 0.43
query29 12.52 3.90 3.25
query30 0.24 0.09 0.06
query31 2.81 0.59 0.38
query32 3.28 0.54 0.46
query33 3.04 2.97 3.03
query34 15.82 5.11 4.48
query35 4.54 4.51 4.51
query36 0.67 0.49 0.50
query37 0.10 0.06 0.06
query38 0.06 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.13 0.13
query41 0.09 0.03 0.03
query42 0.04 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 106.04 s
Total hot run time: 30.57 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2700248814
<details>
<summary>TPC-H: <b>Total hot run time: 32942 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d6b88a7861bafa88a7a6b898d05d0258359e7037, data reload: false
------ Round 1 ----------------------------------
q1 17691 5238 5187 5187
q2 2053 298 169 169
q3 10471 1324 709 709
q4 10240 1034 583 583
q5 7582 2443 2374 2374
q6 198 166 139 139
q7 943 744 612 612
q8 9316 1295 1171 1171
q9 5035 4686 4879 4686
q10 6826 2347 1892 1892
q11 481 281 266 266
q12 356 351 227 227
q13 17755 3719 3112 3112
q14 227 223 214 214
q15 556 496 488 488
q16 652 639 589 589
q17 593 882 352 352
q18 7222 6497 6422 6422
q19 2005 966 565 565
q20 313 324 190 190
q21 2941 2312 2017 2017
q22 1088 1023 978 978
Total cold run time: 104544 ms
Total hot run time: 32942 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5278 5173 5171 5171
q2 249 331 232 232
q3 2198 2696 2289 2289
q4 1495 1826 1396 1396
q5 4303 4160 4149 4149
q6 214 166 131 131
q7 2007 1910 1793 1793
q8 2643 2709 2650 2650
q9 7213 7174 7208 7174
q10 3033 3216 2736 2736
q11 579 503 494 494
q12 711 763 653 653
q13 3515 3872 3219 3219
q14 271 293 292 292
q15 525 490 485 485
q16 658 678 651 651
q17 1193 1673 1337 1337
q18 7827 7668 7403 7403
q19 852 846 829 829
q20 1948 2019 1889 1889
q21 5453 4905 4880 4880
q22 1127 1040 994 994
Total cold run time: 53292 ms
Total hot run time: 50847 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) New algorithm implementation for "match_phrase slop" [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #42932:
URL: https://github.com/apache/doris/pull/42932#discussion_r1822463890
##########
be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query/phrase_matcher.h:
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <CLucene/search/query/TermPositionIterator.h>
Review Comment:
warning: 'CLucene/search/query/TermPositionIterator.h' file not found [clang-diagnostic-error]
```cpp
#include <CLucene/search/query/TermPositionIterator.h>
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
Review Comment:
warning: use of undeclared identifier 'length' [clang-diagnostic-error]
```cpp
void clear() { clear(0, length()); }
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query/phrase_positions.h:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <CLucene/search/query/TermPositionIterator.h>
Review Comment:
warning: 'CLucene/search/query/TermPositionIterator.h' file not found [clang-diagnostic-error]
```cpp
#include <CLucene/search/query/TermPositionIterator.h>
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
+
+ void set(int32_t i) { derived()->set_impl(i); }
Review Comment:
warning: unknown type name 'int32_t' [clang-diagnostic-error]
```cpp
void set(int32_t i) { derived()->set_impl(i); }
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
+
+ void set(int32_t i) { derived()->set_impl(i); }
Review Comment:
warning: use of undeclared identifier 'derived'; did you mean 'Derived'? [clang-diagnostic-error]
```cpp
void set(int32_t i) { derived()->set_impl(i); }
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:23:** 'Derived' declared here
```cpp
template <typename Derived>
^
```
</details>
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
+
+ void set(int32_t i) { derived()->set_impl(i); }
+
+ void clear(int32_t i) { derived()->clear_impl(i); }
+
+ void clear(int32_t start_index, int32_t end_index) { derived()->clear_impl(start_index, end_index); }
Review Comment:
warning: use of undeclared identifier 'derived'; did you mean 'Derived'? [clang-diagnostic-error]
```cpp
void clear(int32_t start_index, int32_t end_index) { derived()->clear_impl(start_index, end_index); }
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:23:** 'Derived' declared here
```cpp
template <typename Derived>
^
```
</details>
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
+
+ void set(int32_t i) { derived()->set_impl(i); }
+
+ void clear(int32_t i) { derived()->clear_impl(i); }
Review Comment:
warning: unknown type name 'int32_t' [clang-diagnostic-error]
```cpp
void clear(int32_t i) { derived()->clear_impl(i); }
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
+
+ void set(int32_t i) { derived()->set_impl(i); }
+
+ void clear(int32_t i) { derived()->clear_impl(i); }
+
+ void clear(int32_t start_index, int32_t end_index) { derived()->clear_impl(start_index, end_index); }
Review Comment:
warning: unknown type name 'int32_t' [clang-diagnostic-error]
```cpp
void clear(int32_t start_index, int32_t end_index) { derived()->clear_impl(start_index, end_index); }
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
Review Comment:
warning: cannot initialize a parameter of type 'int' with an rvalue of type 'std::nullptr_t' [clang-diagnostic-error]
```cpp
PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:24:** passing argument to parameter 'sentinel_object_supplier' here
```cpp
PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
^
```
</details>
##########
be/src/olap/rowset/segment_v2/inverted_index/util/fixed_bit_set.h:
##########
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <memory>
+
+#include "common/exception.h"
+#include "olap/rowset/segment_v2/inverted_index/util/bit_set.h"
+
+namespace doris::segment_v2::inverted_index {
+
+class FixedBitSet : public BitSet<FixedBitSet> {
+public:
+ FixedBitSet(uint32_t num_bits) {
+ _num_bits = num_bits;
+ _bits.resize(bits2words(num_bits));
+ _num_words = _bits.size();
+ }
+
+ bool get(uint32_t index) {
+ assert(index < _num_bits);
+ uint32_t i = index >> 6;
+ uint64_t bit_mask = 1ULL << (index % 64);
+ return (_bits[i] & bit_mask) != 0;
+ }
+
+ void set(uint32_t index) {
+ assert(index < _num_bits);
+ uint32_t word_num = index >> 6;
+ uint64_t bit_mask = 1ULL << (index % 64);
+ _bits[word_num] |= bit_mask;
+ }
+
+ void clear(uint32_t index) {
+ assert(index < _num_bits);
+ uint32_t word_num = index >> 6;
+ uint64_t bit_mask = 1ULL << (index % 64);
+ _bits[word_num] &= ~bit_mask;
+ }
+
+ void clear(uint32_t start_index, uint32_t end_index) {
+ assert(start_index < _num_bits);
+ assert(end_index <= _num_bits);
+ if (end_index <= start_index) {
+ return;
+ }
+
+ uint32_t start_word = start_index >> 6;
+ uint32_t end_word = (end_index - 1) >> 6;
+
+ uint64_t start_mask = ~((1ULL << (start_index % 64)) - 1);
+ uint64_t end_mask = (1ULL << (end_index % 64)) - 1;
+
+ start_mask = ~start_mask;
+ end_mask = ~end_mask;
+
+ if (start_word == end_word) {
+ _bits[start_word] &= (start_mask & end_mask);
+ return;
+ }
+
+ _bits[start_word] &= start_mask;
+ for (uint32_t i = start_word + 1; i < end_word; ++i) {
+ _bits[i] = 0;
+ }
+ _bits[end_word] &= end_mask;
+ }
+
+ void ensure_capacity(size_t num_bits) {
Review Comment:
warning: method 'ensure_capacity' can be made const [readability-make-member-function-const]
```suggestion
void ensure_capacity(size_t num_bits) const {
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
Review Comment:
warning: unknown type name 'size_t' [clang-diagnostic-error]
```cpp
for (size_t i = 1; i < _heap.size(); i++) {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
Review Comment:
warning: use of undeclared identifier 'size_t'; did you mean 'sizeof'? [clang-diagnostic-error]
```suggestion
sizeof index = ++_size;
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
Review Comment:
warning: use of undeclared identifier 'index' [clang-diagnostic-error]
```cpp
size_t index = ++_size;
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query/phrase_positions.h:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <CLucene/search/query/TermPositionIterator.h>
+
+namespace doris::segment_v2::inverted_index {
+
+class PhrasePositions {
+public:
+ PhrasePositions(const TermPositionIterator& postings, int32_t o, int32_t ord,
+ const std::vector<std::string>& terms) {
+ _postings = postings;
+ _offset = o;
+ _ord = ord;
+ _terms = terms;
+ }
+
+ void first_position() {
+ _count = _postings.freq();
+ next_position();
+ }
+
+ bool next_position() {
+ if (_count-- > 0) {
+ _position = _postings.nextPosition() - _offset;
+ return true;
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query/phrase_positions.h:39:
```diff
- if (_count-- > 0) {
- _position = _postings.nextPosition() - _offset;
- return true;
- } else {
- return false;
- }
+ return _count-- > 0;
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
+
+ void set(int32_t i) { derived()->set_impl(i); }
+
+ void clear(int32_t i) { derived()->clear_impl(i); }
+
+ void clear(int32_t start_index, int32_t end_index) { derived()->clear_impl(start_index, end_index); }
Review Comment:
warning: unknown type name 'int32_t' [clang-diagnostic-error]
```cpp
void clear(int32_t start_index, int32_t end_index) { derived()->clear_impl(start_index, end_index); }
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:
##########
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "olap/rowset/segment_v2/inverted_index/util/bits.h"
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class BitSet : public Bits<Derived> {
+public:
+ void clear() { clear(0, length()); }
+
+ void set(int32_t i) { derived()->set_impl(i); }
+
+ void clear(int32_t i) { derived()->clear_impl(i); }
Review Comment:
warning: use of undeclared identifier 'derived'; did you mean 'Derived'? [clang-diagnostic-error]
```cpp
void clear(int32_t i) { derived()->clear_impl(i); }
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/inverted_index/util/bit_set.h:23:** 'Derived' declared here
```cpp
template <typename Derived>
^
```
</details>
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
Review Comment:
warning: unknown type name 'size_t' [clang-diagnostic-error]
```cpp
PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
Review Comment:
warning: use of undeclared identifier 'std' [clang-diagnostic-error]
```cpp
PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bits.h:
##########
@@ -0,0 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class Bits {
+public:
+ bool get(int32_t index) {
+ derived()->get_impl();
+ }
+
+ int32_t length() {
Review Comment:
warning: unknown type name 'int32_t' [clang-diagnostic-error]
```cpp
int32_t length() {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/bits.h:
##########
@@ -0,0 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename Derived>
+class Bits {
+public:
+ bool get(int32_t index) {
Review Comment:
warning: unknown type name 'int32_t' [clang-diagnostic-error]
```cpp
bool get(int32_t index) {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
+ _heap[index] = element;
+ up_heap(index);
Review Comment:
warning: use of undeclared identifier 'index' [clang-diagnostic-error]
```cpp
up_heap(index);
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
Review Comment:
warning: use of undeclared identifier 'heap_size'; did you mean 'max_size'? [clang-diagnostic-error]
```suggestion
size_t max_size = (max_size == 0) ? 2 : max_size + 1;
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:24:** 'max_size' declared here
```cpp
PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
^
```
</details>
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
Review Comment:
warning: use of undeclared identifier 'heap_size' [clang-diagnostic-error]
```cpp
_heap.resize(heap_size);
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
Review Comment:
warning: use of undeclared identifier 'size_t'; did you mean 'sizeof'? [clang-diagnostic-error]
```suggestion
sizeof heap_size = (max_size == 0) ? 2 : max_size + 1;
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
+ _heap[index] = element;
Review Comment:
warning: use of undeclared identifier 'index' [clang-diagnostic-error]
```cpp
_heap[index] = element;
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2791516392
<details>
<summary>TPC-H: <b>Total hot run time: 34191 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 49520d7e9cdd2eda186cfa5815b3db442739314f, data reload: false
------ Round 1 ----------------------------------
q1 25904 5108 5031 5031
q2 2056 272 179 179
q3 10417 1246 702 702
q4 10220 989 525 525
q5 7529 2308 2333 2308
q6 187 163 134 134
q7 920 724 605 605
q8 9321 1303 1119 1119
q9 6864 5150 5140 5140
q10 6871 2312 1890 1890
q11 486 288 280 280
q12 354 355 225 225
q13 17789 3694 3091 3091
q14 225 227 207 207
q15 546 480 477 477
q16 615 616 588 588
q17 585 839 363 363
q18 7581 7089 7080 7080
q19 1315 948 577 577
q20 342 333 218 218
q21 4263 3464 2466 2466
q22 1053 1008 986 986
Total cold run time: 115443 ms
Total hot run time: 34191 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5168 5109 5126 5109
q2 244 327 225 225
q3 2168 2621 2257 2257
q4 1422 1844 1420 1420
q5 4490 4426 4389 4389
q6 214 171 125 125
q7 1999 1899 1759 1759
q8 2617 2493 2536 2493
q9 7349 7134 7012 7012
q10 3015 3269 2733 2733
q11 587 517 501 501
q12 682 765 616 616
q13 3472 3921 3308 3308
q14 313 310 281 281
q15 510 484 480 480
q16 640 687 646 646
q17 1149 1457 1396 1396
q18 7889 7509 7463 7463
q19 820 841 909 841
q20 1926 1975 1839 1839
q21 5374 4994 4804 4804
q22 1105 1060 1049 1049
Total cold run time: 53153 ms
Total hot run time: 50746 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2774702314
<details>
<summary>TPC-DS: <b>Total hot run time: 186907 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f02a3f0f717bfbd6f489653badc6514305e04b9, data reload: false
query1 997 506 488 488
query2 6533 1955 1952 1952
query3 6737 219 220 219
query4 26244 23664 22924 22924
query5 4322 641 471 471
query6 300 213 197 197
query7 4629 494 291 291
query8 304 249 261 249
query9 8571 2586 2604 2586
query10 452 317 270 270
query11 15557 15074 14848 14848
query12 164 112 103 103
query13 1661 506 390 390
query14 8827 6372 6391 6372
query15 208 208 161 161
query16 7009 626 495 495
query17 947 715 587 587
query18 1752 388 292 292
query19 190 184 156 156
query20 118 112 115 112
query21 211 126 113 113
query22 4156 4349 4107 4107
query23 33744 33054 32977 32977
query24 8463 2434 2471 2434
query25 548 472 448 448
query26 1247 270 158 158
query27 2746 508 345 345
query28 4391 2424 2406 2406
query29 770 575 454 454
query30 296 225 190 190
query31 951 847 784 784
query32 78 67 69 67
query33 620 401 327 327
query34 798 857 528 528
query35 810 832 757 757
query36 953 1014 896 896
query37 123 108 84 84
query38 4050 4122 3964 3964
query39 1454 1377 1431 1377
query40 217 123 110 110
query41 55 51 51 51
query42 121 106 104 104
query43 502 512 484 484
query44 1318 801 800 800
query45 181 171 171 171
query46 849 1035 640 640
query47 1735 1788 1732 1732
query48 383 428 318 318
query49 770 493 416 416
query50 646 702 393 393
query51 4197 4110 4085 4085
query52 108 107 104 104
query53 228 265 190 190
query54 582 590 513 513
query55 85 77 81 77
query56 318 298 305 298
query57 1148 1134 1050 1050
query58 272 262 272 262
query59 2741 2766 2931 2766
query60 327 322 297 297
query61 131 129 128 128
query62 803 736 662 662
query63 231 187 197 187
query64 4389 1000 690 690
query65 4316 4276 4276 4276
query66 1162 418 310 310
query67 15786 15541 15304 15304
query68 7954 906 521 521
query69 460 306 267 267
query70 1192 1123 1106 1106
query71 452 326 296 296
query72 5545 4711 4653 4653
query73 645 573 343 343
query74 8832 9165 8758 8758
query75 3812 3178 2697 2697
query76 3612 1192 784 784
query77 796 391 293 293
query78 9818 9995 9355 9355
query79 4365 827 568 568
query80 657 543 432 432
query81 481 263 226 226
query82 586 132 100 100
query83 297 262 242 242
query84 291 115 86 86
query85 794 351 380 351
query86 344 313 291 291
query87 4385 4483 4494 4483
query88 3449 2261 2246 2246
query89 442 379 288 288
query90 1928 217 222 217
query91 142 139 108 108
query92 82 65 57 57
query93 2725 972 587 587
query94 678 409 298 298
query95 379 301 289 289
query96 496 576 286 286
query97 3185 3236 3179 3179
query98 228 216 210 210
query99 1465 1405 1293 1293
Total cold run time: 276049 ms
Total hot run time: 186907 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2775145331
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 merged PR #42932:
URL: https://github.com/apache/doris/pull/42932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2775193886
# BE UT Coverage Report
Increment line coverage `70.60% (778/1102)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1f02a3f0f717bfbd6f489653badc6514305e04b9_1f02a3f0f717bfbd6f489653badc6514305e04b9/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1f02a3f0f717bfbd6f489653badc6514305e04b9_1f02a3f0f717bfbd6f489653badc6514305e04b9/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.32% (14044/26845) |
| Line Coverage | 41.06% (120984/294654) |
| Region Coverage | 39.81% (61605/154741) |
| Branch Coverage | 34.46% (30803/89398) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) New algorithm implementation for "match_phrase slop" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2692593239
<details>
<summary>TPC-H: <b>Total hot run time: 31896 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 937a0a78882b01202fc5726adcd5f90651ff7856, data reload: false
------ Round 1 ----------------------------------
q1 18812 5321 5134 5134
q2 2057 307 166 166
q3 11506 1252 771 771
q4 10210 1030 536 536
q5 7536 2435 2374 2374
q6 196 170 138 138
q7 908 761 609 609
q8 9348 1293 1118 1118
q9 5758 4573 4983 4573
q10 6888 2378 2001 2001
q11 487 294 273 273
q12 352 380 229 229
q13 21792 3801 3148 3148
q14 236 233 209 209
q15 515 467 458 458
q16 681 608 603 603
q17 584 868 348 348
q18 7525 6231 6249 6231
q19 2133 961 552 552
q20 313 312 190 190
q21 2766 2131 1936 1936
q22 361 334 299 299
Total cold run time: 110964 ms
Total hot run time: 31896 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5248 5127 5133 5127
q2 242 329 231 231
q3 2174 2706 2286 2286
q4 1380 1812 1334 1334
q5 4224 4113 4349 4113
q6 219 170 128 128
q7 1995 1914 1780 1780
q8 2660 2665 2572 2572
q9 7411 7265 7382 7265
q10 3054 3203 2716 2716
q11 573 519 505 505
q12 732 808 616 616
q13 3582 3933 3269 3269
q14 296 314 272 272
q15 515 455 460 455
q16 655 696 646 646
q17 1167 1620 1338 1338
q18 7509 7421 7446 7421
q19 810 841 942 841
q20 1994 2005 1877 1877
q21 5421 4942 4870 4870
q22 646 565 545 545
Total cold run time: 52507 ms
Total hot run time: 50207 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2775145249
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) New algorithm implementation for "match_phrase slop" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2692597222
<details>
<summary>TPC-DS: <b>Total hot run time: 189879 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 937a0a78882b01202fc5726adcd5f90651ff7856, data reload: false
query1 1339 930 911 911
query2 6248 1902 1886 1886
query3 11125 4724 4562 4562
query4 25331 23292 22941 22941
query5 3490 668 484 484
query6 298 202 182 182
query7 3985 522 309 309
query8 306 239 223 223
query9 8456 2508 2491 2491
query10 463 314 253 253
query11 15570 14981 15319 14981
query12 183 108 107 107
query13 1546 514 389 389
query14 9309 6239 6516 6239
query15 208 194 181 181
query16 7580 642 493 493
query17 1199 706 568 568
query18 2020 443 342 342
query19 224 213 206 206
query20 126 119 121 119
query21 209 125 105 105
query22 4585 4531 4292 4292
query23 34012 33554 33383 33383
query24 7774 2440 2432 2432
query25 536 475 383 383
query26 1383 281 158 158
query27 2622 479 351 351
query28 4649 2450 2466 2450
query29 725 565 439 439
query30 219 191 155 155
query31 918 893 799 799
query32 78 71 68 68
query33 549 360 308 308
query34 817 856 502 502
query35 790 850 761 761
query36 1043 1022 917 917
query37 120 104 85 85
query38 4376 4483 4128 4128
query39 1501 1478 1418 1418
query40 205 122 103 103
query41 51 50 55 50
query42 127 108 107 107
query43 500 527 487 487
query44 1323 821 829 821
query45 178 173 179 173
query46 891 1056 665 665
query47 1831 1872 1793 1793
query48 398 424 317 317
query49 776 535 424 424
query50 738 754 419 419
query51 4299 4308 4251 4251
query52 126 109 100 100
query53 236 255 195 195
query54 501 508 433 433
query55 92 82 85 82
query56 272 283 255 255
query57 1126 1211 1095 1095
query58 258 262 241 241
query59 2742 2836 2747 2747
query60 305 312 283 283
query61 152 141 138 138
query62 780 758 666 666
query63 237 203 201 201
query64 4445 1023 685 685
query65 3321 3222 3197 3197
query66 1017 409 301 301
query67 16607 15554 15189 15189
query68 9762 878 510 510
query69 475 306 276 276
query70 1201 1104 1111 1104
query71 452 356 267 267
query72 5101 3553 3675 3553
query73 789 709 353 353
query74 8974 9187 8683 8683
query75 4171 3237 2681 2681
query76 4655 1184 762 762
query77 959 455 281 281
query78 9851 10109 9296 9296
query79 1778 842 579 579
query80 644 525 474 474
query81 482 279 249 249
query82 238 125 99 99
query83 209 175 155 155
query84 285 97 76 76
query85 737 354 371 354
query86 329 284 277 277
query87 4594 4554 4395 4395
query88 2873 2253 2199 2199
query89 410 335 293 293
query90 2071 198 196 196
query91 139 142 111 111
query92 76 60 57 57
query93 1218 1043 588 588
query94 667 383 270 270
query95 362 270 259 259
query96 484 561 271 271
query97 3276 3394 3318 3318
query98 218 204 209 204
query99 1409 1391 1270 1270
Total cold run time: 278335 ms
Total hot run time: 189879 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) New algorithm implementation for "match_phrase slop" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2692598965
<details>
<summary>ClickBench: <b>Total hot run time: 31.26 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 937a0a78882b01202fc5726adcd5f90651ff7856, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.03
query3 0.24 0.06 0.07
query4 1.61 0.10 0.10
query5 0.57 0.57 0.56
query6 1.22 0.72 0.73
query7 0.02 0.02 0.02
query8 0.04 0.03 0.03
query9 0.57 0.54 0.54
query10 0.57 0.58 0.57
query11 0.16 0.12 0.11
query12 0.14 0.11 0.12
query13 0.62 0.60 0.59
query14 2.80 2.73 2.69
query15 0.92 0.85 0.85
query16 0.38 0.37 0.40
query17 1.05 1.00 1.06
query18 0.20 0.20 0.20
query19 1.96 2.00 1.85
query20 0.01 0.01 0.01
query21 15.39 0.91 0.55
query22 0.76 1.19 0.59
query23 15.02 1.36 0.61
query24 6.64 1.27 1.21
query25 0.48 0.25 0.12
query26 0.58 0.17 0.14
query27 0.05 0.05 0.05
query28 9.68 0.88 0.42
query29 12.53 4.09 3.36
query30 0.25 0.09 0.07
query31 2.86 0.61 0.38
query32 3.23 0.54 0.46
query33 2.99 3.01 3.07
query34 15.72 5.15 4.57
query35 4.58 4.54 4.48
query36 0.69 0.51 0.48
query37 0.09 0.07 0.06
query38 0.04 0.04 0.04
query39 0.03 0.03 0.02
query40 0.17 0.13 0.12
query41 0.08 0.03 0.03
query42 0.04 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 105.12 s
Total hot run time: 31.26 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2700409043
<details>
<summary>TPC-H: <b>Total hot run time: 32527 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d45a8921e34cb932904b95ebdde40784225b1cd, data reload: false
------ Round 1 ----------------------------------
q1 17573 5204 5044 5044
q2 2046 309 171 171
q3 10395 1257 706 706
q4 10213 1020 535 535
q5 7539 2436 2382 2382
q6 188 170 140 140
q7 925 761 597 597
q8 9309 1272 1134 1134
q9 4964 4829 4904 4829
q10 6813 2294 1879 1879
q11 465 293 255 255
q12 342 351 215 215
q13 17786 3672 3034 3034
q14 239 221 198 198
q15 530 491 477 477
q16 625 621 574 574
q17 589 845 345 345
q18 7110 6526 6333 6333
q19 1080 954 553 553
q20 308 336 198 198
q21 3095 2279 1967 1967
q22 1065 1016 961 961
Total cold run time: 103199 ms
Total hot run time: 32527 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5131 5229 5116 5116
q2 233 327 228 228
q3 2134 2682 2325 2325
q4 1478 1848 1366 1366
q5 4289 4116 4122 4116
q6 210 162 122 122
q7 1893 1949 1769 1769
q8 2573 2534 2607 2534
q9 7197 7141 7303 7141
q10 3032 3214 2733 2733
q11 565 528 509 509
q12 662 761 644 644
q13 3494 3893 3208 3208
q14 293 318 298 298
q15 523 486 465 465
q16 663 694 653 653
q17 1135 1596 1325 1325
q18 7673 7595 7522 7522
q19 823 812 825 812
q20 1963 2002 1908 1908
q21 5497 4887 4890 4887
q22 1089 1081 998 998
Total cold run time: 52550 ms
Total hot run time: 50679 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2702731260
# BE UT Coverage Report
Increment line coverage `42.15% (778/1846)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2bb72e3e7db4f83f62843d2461874e1a885b5419_2bb72e3e7db4f83f62843d2461874e1a885b5419/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2bb72e3e7db4f83f62843d2461874e1a885b5419_2bb72e3e7db4f83f62843d2461874e1a885b5419/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 46.05% (12340/26796) |
| Line Coverage | 35.51% (104213/293483) |
| Region Coverage | 34.70% (53429/153991) |
| Branch Coverage | 30.40% (27086/89110) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2702566191
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on code in PR #42932:
URL: https://github.com/apache/doris/pull/42932#discussion_r2011386784
##########
be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query/exact_phrase_matcher.cpp:
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/inverted_index/query/phrase_query/exact_phrase_matcher.h"
+
+namespace doris::segment_v2::inverted_index {
+
+ExactPhraseMatcher::ExactPhraseMatcher(std::vector<PostingsAndPosition> postings)
Review Comment:
Use code below to avoid params copy when call ExactPhraseMatcher constructor
`class ExactPhraseMatcher {
public:
ExactPhraseMatcher(const std::vector<PostingsAndPosition>& postings)
: _postings(postings) {}
ExactPhraseMatcher(std::vector<PostingsAndPosition>&& postings)
: _postings(std::move(postings)) {}
};`
##########
be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query/exact_phrase_matcher.cpp:
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/inverted_index/query/phrase_query/exact_phrase_matcher.h"
+
+namespace doris::segment_v2::inverted_index {
+
+ExactPhraseMatcher::ExactPhraseMatcher(std::vector<PostingsAndPosition> postings)
Review Comment:
Use code below to avoid params copy when call ExactPhraseMatcher constructor
```class ExactPhraseMatcher {
public:
ExactPhraseMatcher(const std::vector<PostingsAndPosition>& postings)
: _postings(postings) {}
ExactPhraseMatcher(std::vector<PostingsAndPosition>&& postings)
: _postings(std::move(postings)) {}
};```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2791476484
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2700153603
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2700275253
<details>
<summary>TPC-DS: <b>Total hot run time: 186204 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d6b88a7861bafa88a7a6b898d05d0258359e7037, data reload: false
query1 1009 396 372 372
query2 6536 1870 1791 1791
query3 6805 227 216 216
query4 26862 23192 22961 22961
query5 4364 667 486 486
query6 298 209 192 192
query7 4606 502 291 291
query8 299 237 231 231
query9 8603 2509 2530 2509
query10 468 324 269 269
query11 15234 15218 15100 15100
query12 164 114 108 108
query13 1659 522 397 397
query14 9071 6823 6789 6789
query15 214 207 177 177
query16 7195 643 475 475
query17 1185 714 546 546
query18 1956 398 296 296
query19 190 177 164 164
query20 122 113 117 113
query21 211 120 99 99
query22 4267 4379 4484 4379
query23 33852 32996 33053 32996
query24 7768 2383 2445 2383
query25 548 453 396 396
query26 1232 269 153 153
query27 2157 516 325 325
query28 3919 2437 2378 2378
query29 761 557 421 421
query30 290 212 196 196
query31 937 836 768 768
query32 75 62 58 58
query33 548 354 301 301
query34 782 861 506 506
query35 810 832 735 735
query36 978 1003 882 882
query37 109 98 78 78
query38 4168 4202 4138 4138
query39 1461 1412 1386 1386
query40 217 128 120 120
query41 61 56 57 56
query42 136 113 113 113
query43 496 507 475 475
query44 1290 812 808 808
query45 179 179 168 168
query46 856 1032 622 622
query47 1769 1792 1701 1701
query48 381 437 330 330
query49 792 523 425 425
query50 680 738 408 408
query51 4149 4143 4127 4127
query52 104 108 90 90
query53 232 259 181 181
query54 499 502 399 399
query55 85 81 81 81
query56 265 255 240 240
query57 1130 1160 1066 1066
query58 248 250 236 236
query59 2673 2675 2457 2457
query60 288 275 275 275
query61 121 117 150 117
query62 812 732 664 664
query63 229 189 193 189
query64 4360 996 660 660
query65 4380 4309 4312 4309
query66 1131 410 294 294
query67 15954 15536 15501 15501
query68 8001 876 519 519
query69 486 312 268 268
query70 1208 1167 1097 1097
query71 409 298 265 265
query72 5660 3643 3738 3643
query73 766 755 338 338
query74 8943 8939 8989 8939
query75 3203 3186 2695 2695
query76 3172 1177 835 835
query77 502 373 292 292
query78 9824 10007 9415 9415
query79 1587 832 579 579
query80 670 536 456 456
query81 494 262 219 219
query82 208 125 99 99
query83 177 174 152 152
query84 243 103 72 72
query85 802 354 312 312
query86 370 311 309 309
query87 4470 4527 4562 4527
query88 2824 2179 2156 2156
query89 389 317 282 282
query90 1964 199 195 195
query91 137 146 110 110
query92 79 61 58 58
query93 1665 1060 577 577
query94 679 375 302 302
query95 366 270 263 263
query96 491 556 267 267
query97 3263 3431 3296 3296
query98 230 218 206 206
query99 1298 1401 1288 1288
Total cold run time: 269810 ms
Total hot run time: 186204 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2702672783
<details>
<summary>TPC-DS: <b>Total hot run time: 192698 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2bb72e3e7db4f83f62843d2461874e1a885b5419, data reload: false
query1 1413 1028 993 993
query2 6385 1866 1826 1826
query3 11155 4655 4588 4588
query4 54849 24918 23742 23742
query5 5112 545 500 500
query6 383 206 209 206
query7 5024 494 294 294
query8 319 241 246 241
query9 6077 2549 2573 2549
query10 412 321 258 258
query11 15284 15004 14880 14880
query12 159 115 113 113
query13 1119 513 392 392
query14 10592 6860 6507 6507
query15 214 196 181 181
query16 7084 690 526 526
query17 1101 735 622 622
query18 1559 429 335 335
query19 212 195 164 164
query20 129 126 121 121
query21 257 130 103 103
query22 4582 4521 4540 4521
query23 34086 33375 33527 33375
query24 5789 2410 2447 2410
query25 458 470 400 400
query26 724 279 169 169
query27 1776 493 341 341
query28 2755 2447 2412 2412
query29 576 565 437 437
query30 284 220 198 198
query31 873 897 791 791
query32 71 61 62 61
query33 451 359 300 300
query34 781 874 500 500
query35 786 832 774 774
query36 946 999 897 897
query37 116 101 78 78
query38 4243 4230 4244 4230
query39 1478 1424 1435 1424
query40 205 127 100 100
query41 52 50 48 48
query42 122 114 100 100
query43 491 513 493 493
query44 1322 799 794 794
query45 184 174 162 162
query46 853 1040 656 656
query47 1846 1902 1763 1763
query48 398 426 309 309
query49 694 494 409 409
query50 726 754 413 413
query51 4334 4335 4230 4230
query52 111 96 101 96
query53 225 253 190 190
query54 497 486 415 415
query55 87 82 76 76
query56 256 310 270 270
query57 1190 1197 1123 1123
query58 249 247 230 230
query59 2988 2792 2637 2637
query60 278 284 253 253
query61 119 134 115 115
query62 777 778 693 693
query63 234 197 188 188
query64 1790 1067 726 726
query65 4563 4377 4329 4329
query66 717 392 296 296
query67 16007 15522 15442 15442
query68 7091 819 501 501
query69 539 302 281 281
query70 1179 1071 1098 1071
query71 505 316 266 266
query72 5742 3711 3712 3711
query73 1385 745 351 351
query74 8960 8983 8973 8973
query75 3824 3150 2710 2710
query76 4228 1167 740 740
query77 617 363 271 271
query78 10063 10165 9271 9271
query79 2383 830 607 607
query80 600 509 450 450
query81 480 251 217 217
query82 485 126 96 96
query83 174 164 158 158
query84 296 93 71 71
query85 798 351 303 303
query86 380 316 270 270
query87 4586 4483 4452 4452
query88 3718 2181 2149 2149
query89 408 319 278 278
query90 1763 193 231 193
query91 132 138 106 106
query92 81 61 58 58
query93 1891 1065 577 577
query94 642 394 267 267
query95 356 266 263 263
query96 494 553 266 266
query97 3332 3394 3275 3275
query98 233 202 203 202
query99 1323 1428 1279 1279
Total cold run time: 299617 ms
Total hot run time: 192698 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2862572186
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) New algorithm implementation for "match_phrase slop" [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2692583370
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2700361247
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2702678338
<details>
<summary>ClickBench: <b>Total hot run time: 30.73 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2bb72e3e7db4f83f62843d2461874e1a885b5419, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.04 0.04
query3 0.25 0.06 0.06
query4 1.62 0.10 0.10
query5 0.56 0.54 0.54
query6 1.21 0.72 0.72
query7 0.02 0.01 0.02
query8 0.04 0.04 0.04
query9 0.59 0.53 0.52
query10 0.58 0.58 0.58
query11 0.16 0.10 0.11
query12 0.14 0.11 0.11
query13 0.62 0.59 0.60
query14 2.81 2.70 2.68
query15 0.92 0.84 0.84
query16 0.37 0.39 0.37
query17 1.01 1.03 1.03
query18 0.22 0.20 0.20
query19 1.88 1.80 1.96
query20 0.02 0.01 0.02
query21 15.37 0.90 0.54
query22 0.75 1.16 0.61
query23 15.01 1.38 0.63
query24 6.80 1.91 0.78
query25 0.50 0.26 0.08
query26 0.63 0.17 0.15
query27 0.06 0.05 0.04
query28 9.27 0.86 0.44
query29 12.57 3.90 3.27
query30 0.25 0.09 0.07
query31 2.82 0.59 0.39
query32 3.23 0.56 0.46
query33 2.99 3.01 2.99
query34 15.73 5.11 4.56
query35 4.59 4.60 4.54
query36 0.67 0.50 0.48
query37 0.09 0.07 0.06
query38 0.06 0.04 0.04
query39 0.03 0.03 0.03
query40 0.16 0.14 0.13
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 104.87 s
Total hot run time: 30.73 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2701341220
<details>
<summary>TPC-H: <b>Total hot run time: 32574 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4bf31488c6c90a028a6d62a484c8bc8a1f2b5539, data reload: false
------ Round 1 ----------------------------------
q1 17587 5281 5081 5081
q2 2061 312 169 169
q3 10547 1237 754 754
q4 10217 1023 565 565
q5 7538 2354 2344 2344
q6 187 168 135 135
q7 907 777 608 608
q8 9314 1283 1101 1101
q9 4941 4826 4721 4721
q10 6860 2319 1902 1902
q11 519 276 264 264
q12 347 361 226 226
q13 17771 3695 3101 3101
q14 227 222 208 208
q15 542 487 479 479
q16 624 612 597 597
q17 597 858 346 346
q18 6898 6513 6301 6301
q19 1995 949 550 550
q20 316 324 186 186
q21 2978 2169 1949 1949
q22 1069 1015 987 987
Total cold run time: 104042 ms
Total hot run time: 32574 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5277 5150 5101 5101
q2 234 340 231 231
q3 2193 2672 2267 2267
q4 1460 1851 1399 1399
q5 4225 4151 4150 4150
q6 201 160 123 123
q7 1899 1964 1762 1762
q8 2620 2662 2604 2604
q9 7229 7232 7184 7184
q10 3068 3193 2767 2767
q11 567 500 508 500
q12 658 746 588 588
q13 3561 3978 3269 3269
q14 294 288 266 266
q15 515 483 501 483
q16 656 701 636 636
q17 1166 1606 1348 1348
q18 7735 7618 7606 7606
q19 814 813 866 813
q20 1966 2028 1896 1896
q21 5599 4924 4784 4784
q22 1082 1097 1019 1019
Total cold run time: 53019 ms
Total hot run time: 50796 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) New algorithm implementation for "match_phrase slop" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2446822768
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) New algorithm implementation for "match_phrase slop" [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #42932:
URL: https://github.com/apache/doris/pull/42932#discussion_r1824284907
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
+ _heap[index] = element;
+ up_heap(index);
+ return _heap[1];
+ }
+
+ T insert_with_overflow(const T& element) {
+ if (_size < _max_size) {
+ add(element);
+ return T();
+ } else if (_size > 0 && less_than(_heap[1], element)) {
+ T ret = _heap[1];
+ _heap[1] = element;
+ update_top();
+ return ret;
+ } else {
+ return element;
+ }
+ }
+
+ const T& top() const { return _heap[1]; }
+
+ T pop() {
+ if (_size > 0) {
+ T result = _heap[1];
+ _heap[1] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ down_heap(1);
+ return result;
+ }
+ return T();
+ }
+
+ T update_top() {
+ down_heap(1);
+ return _heap[1];
+ }
+
+ T update_top(const T& new_top) {
+ _heap[1] = new_top;
+ return update_top();
+ }
+
+ void clear() {
+ for (size_t i = 1; i <= _size; i++) {
+ _heap[i] = T();
+ }
+ _size = 0;
+ }
+
+ bool remove(const T& element) {
+ for (size_t i = 1; i <= _size; i++) {
+ if (_heap[i] == element) {
+ _heap[i] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ if (i <= _size) {
+ if (!up_heap(i)) {
+ down_heap(i);
+ }
+ }
+ return true;
+ }
+ }
+ return false;
+ }
+
+private:
+ virtual bool less_than(T a, T b) const = 0;
+
+ bool up_heap(size_t orig_pos) {
+ size_t i = orig_pos;
+ T node = _heap[i];
+ size_t j = i >> 2;
+ while (j > 0 && less_than(node, _heap[j])) {
+ _heap[i] = _heap[j];
+ i = j;
+ j = j >> 2;
+ }
+ _heap[i] = node;
+ return i != orig_pos;
+ }
+
+ void down_heap(size_t i) {
+ T node = _heap[i];
+ size_t j = i << 2;
+ size_t k = j + 1;
+ if (k <= _size && less_than(_heap[k], _heap[j])) {
+ j = k;
+ }
+ while (j <= _size && less_than(_heap[j], node)) {
+ _heap[i] = _heap[j];
+ i = j;
+ j = i << 2;
+ k = j + 1;
+ if (k <= _size && less_than(_heap[k], _heap[j])) {
+ j = k;
+ }
+ }
+ _heap[i] = node;
+ }
+
+ size_t _size = 0;
Review Comment:
warning: unknown type name 'size_t' [clang-diagnostic-error]
```cpp
size_t _size = 0;
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query/sloppy_phrase_matcher.h:
##########
@@ -0,0 +1,337 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <limits>
+#include <memory>
+
+#include "common/exception.h"
+#include "olap/rowset/segment_v2/inverted_index/query/phrase_query/phrase_matcher.h"
+#include "olap/rowset/segment_v2/inverted_index/query/phrase_query/phrase_positions.h"
+#include "olap/rowset/segment_v2/inverted_index/query/phrase_query/phrase_queue.h"
+#include "olap/rowset/segment_v2/inverted_index/util/fixed_bit_set.h"
+
+namespace doris::segment_v2::inverted_index {
+
+class SloppyPhraseMatcher : public PhraseMatcherBase<SloppyPhraseMatcher> {
+public:
+ SloppyPhraseMatcher(const std::vector<PostingsAndFreq>& postings, int32_t slop) {
+ _slop = slop;
+ _num_postings = postings.size();
+ _pq = std::make_unique<PhraseQueue>(postings.size());
+ _phrase_positions.resize(postings.size());
+ for (size_t i = 0; i < postings.size(); i++) {
+ _phrase_positions[i] = new PhrasePositions(postings[i]._postings, postings[i]._position,
+ i, postings[i]._terms);
+ }
+ }
+
+ ~SloppyPhraseMatcher() override {
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```cpp
~SloppyPhraseMatcher() override {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
+ _heap[index] = element;
+ up_heap(index);
+ return _heap[1];
+ }
+
+ T insert_with_overflow(const T& element) {
+ if (_size < _max_size) {
+ add(element);
+ return T();
+ } else if (_size > 0 && less_than(_heap[1], element)) {
+ T ret = _heap[1];
+ _heap[1] = element;
+ update_top();
+ return ret;
+ } else {
+ return element;
+ }
+ }
+
+ const T& top() const { return _heap[1]; }
+
+ T pop() {
+ if (_size > 0) {
+ T result = _heap[1];
+ _heap[1] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ down_heap(1);
+ return result;
+ }
+ return T();
+ }
+
+ T update_top() {
+ down_heap(1);
+ return _heap[1];
+ }
+
+ T update_top(const T& new_top) {
+ _heap[1] = new_top;
+ return update_top();
+ }
+
+ void clear() {
+ for (size_t i = 1; i <= _size; i++) {
+ _heap[i] = T();
+ }
+ _size = 0;
+ }
+
+ bool remove(const T& element) {
+ for (size_t i = 1; i <= _size; i++) {
+ if (_heap[i] == element) {
+ _heap[i] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ if (i <= _size) {
+ if (!up_heap(i)) {
+ down_heap(i);
+ }
+ }
+ return true;
+ }
+ }
+ return false;
+ }
+
+private:
+ virtual bool less_than(T a, T b) const = 0;
+
+ bool up_heap(size_t orig_pos) {
+ size_t i = orig_pos;
Review Comment:
warning: use of undeclared identifier 'size_t'; did you mean 'sizeof'? [clang-diagnostic-error]
```suggestion
sizeof i = orig_pos;
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
+ _heap[index] = element;
+ up_heap(index);
+ return _heap[1];
+ }
+
+ T insert_with_overflow(const T& element) {
+ if (_size < _max_size) {
+ add(element);
+ return T();
+ } else if (_size > 0 && less_than(_heap[1], element)) {
+ T ret = _heap[1];
+ _heap[1] = element;
+ update_top();
+ return ret;
+ } else {
+ return element;
+ }
+ }
+
+ const T& top() const { return _heap[1]; }
+
+ T pop() {
+ if (_size > 0) {
+ T result = _heap[1];
+ _heap[1] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ down_heap(1);
+ return result;
+ }
+ return T();
+ }
+
+ T update_top() {
+ down_heap(1);
+ return _heap[1];
+ }
+
+ T update_top(const T& new_top) {
+ _heap[1] = new_top;
+ return update_top();
+ }
+
+ void clear() {
+ for (size_t i = 1; i <= _size; i++) {
Review Comment:
warning: unknown type name 'size_t' [clang-diagnostic-error]
```cpp
for (size_t i = 1; i <= _size; i++) {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
+ _heap[index] = element;
+ up_heap(index);
+ return _heap[1];
+ }
+
+ T insert_with_overflow(const T& element) {
+ if (_size < _max_size) {
+ add(element);
+ return T();
+ } else if (_size > 0 && less_than(_heap[1], element)) {
+ T ret = _heap[1];
+ _heap[1] = element;
+ update_top();
+ return ret;
+ } else {
+ return element;
+ }
+ }
+
+ const T& top() const { return _heap[1]; }
+
+ T pop() {
+ if (_size > 0) {
+ T result = _heap[1];
+ _heap[1] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ down_heap(1);
+ return result;
+ }
+ return T();
+ }
+
+ T update_top() {
+ down_heap(1);
+ return _heap[1];
+ }
+
+ T update_top(const T& new_top) {
+ _heap[1] = new_top;
+ return update_top();
+ }
+
+ void clear() {
+ for (size_t i = 1; i <= _size; i++) {
+ _heap[i] = T();
+ }
+ _size = 0;
+ }
+
+ bool remove(const T& element) {
+ for (size_t i = 1; i <= _size; i++) {
+ if (_heap[i] == element) {
+ _heap[i] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ if (i <= _size) {
+ if (!up_heap(i)) {
+ down_heap(i);
+ }
+ }
+ return true;
+ }
+ }
+ return false;
+ }
+
+private:
+ virtual bool less_than(T a, T b) const = 0;
+
+ bool up_heap(size_t orig_pos) {
Review Comment:
warning: unknown type name 'size_t' [clang-diagnostic-error]
```cpp
bool up_heap(size_t orig_pos) {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
+ _heap[index] = element;
+ up_heap(index);
+ return _heap[1];
+ }
+
+ T insert_with_overflow(const T& element) {
+ if (_size < _max_size) {
+ add(element);
+ return T();
+ } else if (_size > 0 && less_than(_heap[1], element)) {
+ T ret = _heap[1];
+ _heap[1] = element;
+ update_top();
+ return ret;
+ } else {
+ return element;
+ }
+ }
+
+ const T& top() const { return _heap[1]; }
+
+ T pop() {
+ if (_size > 0) {
+ T result = _heap[1];
+ _heap[1] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ down_heap(1);
+ return result;
+ }
+ return T();
+ }
+
+ T update_top() {
+ down_heap(1);
+ return _heap[1];
+ }
+
+ T update_top(const T& new_top) {
+ _heap[1] = new_top;
+ return update_top();
+ }
+
+ void clear() {
+ for (size_t i = 1; i <= _size; i++) {
+ _heap[i] = T();
+ }
+ _size = 0;
+ }
+
+ bool remove(const T& element) {
+ for (size_t i = 1; i <= _size; i++) {
+ if (_heap[i] == element) {
+ _heap[i] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ if (i <= _size) {
+ if (!up_heap(i)) {
+ down_heap(i);
+ }
+ }
+ return true;
+ }
+ }
+ return false;
+ }
+
+private:
+ virtual bool less_than(T a, T b) const = 0;
+
+ bool up_heap(size_t orig_pos) {
+ size_t i = orig_pos;
+ T node = _heap[i];
+ size_t j = i >> 2;
+ while (j > 0 && less_than(node, _heap[j])) {
+ _heap[i] = _heap[j];
+ i = j;
+ j = j >> 2;
+ }
+ _heap[i] = node;
+ return i != orig_pos;
+ }
+
+ void down_heap(size_t i) {
+ T node = _heap[i];
+ size_t j = i << 2;
+ size_t k = j + 1;
+ if (k <= _size && less_than(_heap[k], _heap[j])) {
+ j = k;
+ }
+ while (j <= _size && less_than(_heap[j], node)) {
+ _heap[i] = _heap[j];
+ i = j;
+ j = i << 2;
+ k = j + 1;
+ if (k <= _size && less_than(_heap[k], _heap[j])) {
+ j = k;
+ }
+ }
+ _heap[i] = node;
+ }
+
+ size_t _size = 0;
+ size_t _max_size = 0;
Review Comment:
warning: unknown type name 'size_t' [clang-diagnostic-error]
```cpp
size_t _max_size = 0;
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
+ _heap[index] = element;
+ up_heap(index);
+ return _heap[1];
+ }
+
+ T insert_with_overflow(const T& element) {
+ if (_size < _max_size) {
+ add(element);
+ return T();
+ } else if (_size > 0 && less_than(_heap[1], element)) {
+ T ret = _heap[1];
+ _heap[1] = element;
+ update_top();
+ return ret;
+ } else {
+ return element;
+ }
+ }
+
+ const T& top() const { return _heap[1]; }
+
+ T pop() {
+ if (_size > 0) {
+ T result = _heap[1];
+ _heap[1] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ down_heap(1);
+ return result;
+ }
+ return T();
+ }
+
+ T update_top() {
+ down_heap(1);
+ return _heap[1];
+ }
+
+ T update_top(const T& new_top) {
+ _heap[1] = new_top;
+ return update_top();
+ }
+
+ void clear() {
+ for (size_t i = 1; i <= _size; i++) {
+ _heap[i] = T();
+ }
+ _size = 0;
+ }
+
+ bool remove(const T& element) {
+ for (size_t i = 1; i <= _size; i++) {
+ if (_heap[i] == element) {
+ _heap[i] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ if (i <= _size) {
+ if (!up_heap(i)) {
+ down_heap(i);
+ }
+ }
+ return true;
+ }
+ }
+ return false;
+ }
+
+private:
+ virtual bool less_than(T a, T b) const = 0;
+
+ bool up_heap(size_t orig_pos) {
+ size_t i = orig_pos;
+ T node = _heap[i];
+ size_t j = i >> 2;
+ while (j > 0 && less_than(node, _heap[j])) {
+ _heap[i] = _heap[j];
+ i = j;
+ j = j >> 2;
+ }
+ _heap[i] = node;
+ return i != orig_pos;
+ }
+
+ void down_heap(size_t i) {
+ T node = _heap[i];
+ size_t j = i << 2;
+ size_t k = j + 1;
+ if (k <= _size && less_than(_heap[k], _heap[j])) {
+ j = k;
+ }
+ while (j <= _size && less_than(_heap[j], node)) {
+ _heap[i] = _heap[j];
+ i = j;
+ j = i << 2;
+ k = j + 1;
+ if (k <= _size && less_than(_heap[k], _heap[j])) {
+ j = k;
+ }
+ }
+ _heap[i] = node;
+ }
+
+ size_t _size = 0;
+ size_t _max_size = 0;
+ std::vector<T> _heap;
Review Comment:
warning: use of undeclared identifier 'std' [clang-diagnostic-error]
```cpp
std::vector<T> _heap;
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
+ _heap[index] = element;
+ up_heap(index);
+ return _heap[1];
+ }
+
+ T insert_with_overflow(const T& element) {
+ if (_size < _max_size) {
+ add(element);
+ return T();
+ } else if (_size > 0 && less_than(_heap[1], element)) {
+ T ret = _heap[1];
+ _heap[1] = element;
+ update_top();
+ return ret;
+ } else {
+ return element;
+ }
+ }
+
+ const T& top() const { return _heap[1]; }
+
+ T pop() {
+ if (_size > 0) {
+ T result = _heap[1];
+ _heap[1] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ down_heap(1);
+ return result;
+ }
+ return T();
+ }
+
+ T update_top() {
+ down_heap(1);
+ return _heap[1];
+ }
+
+ T update_top(const T& new_top) {
+ _heap[1] = new_top;
+ return update_top();
+ }
+
+ void clear() {
+ for (size_t i = 1; i <= _size; i++) {
+ _heap[i] = T();
+ }
+ _size = 0;
+ }
+
+ bool remove(const T& element) {
+ for (size_t i = 1; i <= _size; i++) {
Review Comment:
warning: unknown type name 'size_t' [clang-diagnostic-error]
```cpp
for (size_t i = 1; i <= _size; i++) {
^
```
##########
be/src/olap/rowset/segment_v2/inverted_index/util/priority_queue.h:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+namespace doris::segment_v2::inverted_index {
+
+template <typename T>
+class PriorityQueue {
+public:
+ PriorityQueue(size_t max_size, std::function<T()> sentinel_object_supplier = nullptr) {
+ size_t heap_size = (max_size == 0) ? 2 : max_size + 1;
+ _heap.resize(heap_size);
+ _max_size = max_size;
+
+ if (sentinel_object_supplier) {
+ for (size_t i = 1; i < _heap.size(); i++) {
+ _heap[i] = sentinel_object_supplier();
+ }
+ _size = max_size;
+ }
+ }
+
+ virtual ~PriorityQueue() = default;
+
+ T add(const T& element) {
+ assert(_size + 1 <= _max_size);
+ size_t index = ++_size;
+ _heap[index] = element;
+ up_heap(index);
+ return _heap[1];
+ }
+
+ T insert_with_overflow(const T& element) {
+ if (_size < _max_size) {
+ add(element);
+ return T();
+ } else if (_size > 0 && less_than(_heap[1], element)) {
+ T ret = _heap[1];
+ _heap[1] = element;
+ update_top();
+ return ret;
+ } else {
+ return element;
+ }
+ }
+
+ const T& top() const { return _heap[1]; }
+
+ T pop() {
+ if (_size > 0) {
+ T result = _heap[1];
+ _heap[1] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ down_heap(1);
+ return result;
+ }
+ return T();
+ }
+
+ T update_top() {
+ down_heap(1);
+ return _heap[1];
+ }
+
+ T update_top(const T& new_top) {
+ _heap[1] = new_top;
+ return update_top();
+ }
+
+ void clear() {
+ for (size_t i = 1; i <= _size; i++) {
+ _heap[i] = T();
+ }
+ _size = 0;
+ }
+
+ bool remove(const T& element) {
+ for (size_t i = 1; i <= _size; i++) {
+ if (_heap[i] == element) {
+ _heap[i] = _heap[_size];
+ _heap[_size] = T();
+ _size--;
+ if (i <= _size) {
+ if (!up_heap(i)) {
+ down_heap(i);
+ }
+ }
+ return true;
+ }
+ }
+ return false;
+ }
+
+private:
+ virtual bool less_than(T a, T b) const = 0;
+
+ bool up_heap(size_t orig_pos) {
+ size_t i = orig_pos;
+ T node = _heap[i];
+ size_t j = i >> 2;
+ while (j > 0 && less_than(node, _heap[j])) {
+ _heap[i] = _heap[j];
+ i = j;
+ j = j >> 2;
+ }
+ _heap[i] = node;
+ return i != orig_pos;
+ }
+
+ void down_heap(size_t i) {
Review Comment:
warning: unknown type name 'size_t' [clang-diagnostic-error]
```cpp
void down_heap(size_t i) {
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on code in PR #42932:
URL: https://github.com/apache/doris/pull/42932#discussion_r2011386784
##########
be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query/exact_phrase_matcher.cpp:
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/inverted_index/query/phrase_query/exact_phrase_matcher.h"
+
+namespace doris::segment_v2::inverted_index {
+
+ExactPhraseMatcher::ExactPhraseMatcher(std::vector<PostingsAndPosition> postings)
Review Comment:
Use code below to avoid params copy when call ExactPhraseMatcher constructor
```class ExactPhraseMatcher {
public:
ExactPhraseMatcher(const std::vector<PostingsAndPosition>& postings)
: _postings(postings) {}
ExactPhraseMatcher(std::vector<PostingsAndPosition>&& postings)
: _postings(std::move(postings)) {}
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2774641406
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2700438594
<details>
<summary>TPC-DS: <b>Total hot run time: 184037 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d45a8921e34cb932904b95ebdde40784225b1cd, data reload: false
query1 996 374 401 374
query2 6521 1811 1861 1811
query3 6794 210 212 210
query4 26375 24001 22917 22917
query5 5080 694 479 479
query6 307 206 185 185
query7 4610 496 294 294
query8 303 238 238 238
query9 8658 2543 2537 2537
query10 502 319 257 257
query11 15666 15397 14904 14904
query12 171 110 101 101
query13 1634 513 374 374
query14 9989 6120 6006 6006
query15 208 188 178 178
query16 7637 668 504 504
query17 1163 729 574 574
query18 1989 416 311 311
query19 200 193 159 159
query20 123 121 113 113
query21 214 125 104 104
query22 4184 4363 3989 3989
query23 33906 32833 33031 32833
query24 7731 2373 2415 2373
query25 559 489 415 415
query26 1235 273 158 158
query27 2584 471 332 332
query28 4304 2483 2408 2408
query29 750 569 440 440
query30 283 222 191 191
query31 988 832 786 786
query32 76 66 61 61
query33 554 360 298 298
query34 783 832 492 492
query35 789 816 721 721
query36 949 976 898 898
query37 122 98 84 84
query38 4083 4322 4202 4202
query39 1457 1404 1368 1368
query40 206 118 101 101
query41 53 49 49 49
query42 125 112 100 100
query43 482 506 463 463
query44 1254 782 769 769
query45 171 164 164 164
query46 838 1008 609 609
query47 1742 1773 1711 1711
query48 370 403 299 299
query49 781 494 426 426
query50 688 720 416 416
query51 4166 4214 4133 4133
query52 109 102 89 89
query53 232 252 180 180
query54 483 491 411 411
query55 89 82 78 78
query56 280 271 250 250
query57 1120 1131 1062 1062
query58 242 245 257 245
query59 2710 2702 2637 2637
query60 295 285 251 251
query61 125 120 115 115
query62 803 732 650 650
query63 230 194 187 187
query64 4294 1027 659 659
query65 4404 4309 4353 4309
query66 1037 418 310 310
query67 15671 15531 15460 15460
query68 8441 916 504 504
query69 511 295 254 254
query70 1189 1133 1016 1016
query71 451 299 279 279
query72 5272 3540 3743 3540
query73 790 714 350 350
query74 9258 9060 8700 8700
query75 3818 3169 2705 2705
query76 3732 1167 744 744
query77 777 366 279 279
query78 9977 10288 9361 9361
query79 2377 830 580 580
query80 626 513 449 449
query81 472 264 221 221
query82 682 124 96 96
query83 172 168 161 161
query84 245 102 73 73
query85 757 367 384 367
query86 337 321 291 291
query87 4345 4630 4285 4285
query88 3362 2161 2146 2146
query89 388 372 291 291
query90 1981 193 195 193
query91 140 154 110 110
query92 76 59 58 58
query93 1251 1044 587 587
query94 678 403 306 306
query95 341 265 264 264
query96 480 556 267 267
query97 3314 3400 3299 3299
query98 223 199 203 199
query99 1676 1387 1250 1250
Total cold run time: 275616 ms
Total hot run time: 184037 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2702549790
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2702656199
<details>
<summary>TPC-H: <b>Total hot run time: 32751 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2bb72e3e7db4f83f62843d2461874e1a885b5419, data reload: false
------ Round 1 ----------------------------------
q1 17600 5287 5120 5120
q2 2056 294 175 175
q3 10597 1231 735 735
q4 10191 1011 543 543
q5 7506 2360 2353 2353
q6 189 165 134 134
q7 884 764 591 591
q8 9294 1234 1123 1123
q9 4984 4829 4877 4829
q10 6854 2300 1895 1895
q11 484 302 262 262
q12 346 353 219 219
q13 17754 3624 3049 3049
q14 239 221 203 203
q15 536 504 491 491
q16 629 622 586 586
q17 576 858 345 345
q18 6788 6524 6405 6405
q19 1617 938 548 548
q20 309 319 184 184
q21 2850 2173 1927 1927
q22 1091 1060 1034 1034
Total cold run time: 103374 ms
Total hot run time: 32751 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5147 5112 5180 5112
q2 250 333 240 240
q3 2275 2740 2424 2424
q4 1477 1841 1410 1410
q5 4255 4119 4127 4119
q6 210 160 125 125
q7 1883 1957 1782 1782
q8 2639 2607 2515 2515
q9 7292 7229 7229 7229
q10 3065 3239 2787 2787
q11 562 519 496 496
q12 644 724 570 570
q13 3503 3854 3268 3268
q14 289 290 286 286
q15 534 488 470 470
q16 657 674 642 642
q17 1153 1576 1389 1389
q18 7741 7618 7489 7489
q19 809 797 896 797
q20 1987 2036 1894 1894
q21 5561 4952 4724 4724
q22 1140 1046 1027 1027
Total cold run time: 53073 ms
Total hot run time: 50795 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on code in PR #42932:
URL: https://github.com/apache/doris/pull/42932#discussion_r2020517194
##########
be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query/exact_phrase_matcher.cpp:
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/inverted_index/query/phrase_query/exact_phrase_matcher.h"
+
+namespace doris::segment_v2::inverted_index {
+
+ExactPhraseMatcher::ExactPhraseMatcher(std::vector<PostingsAndPosition> postings)
Review Comment:
The first approach (pass-by-value + std::move) avoids parameter copying through two move operations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2774713064
<details>
<summary>ClickBench: <b>Total hot run time: 31.67 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1f02a3f0f717bfbd6f489653badc6514305e04b9, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.10 0.11
query3 0.24 0.19 0.19
query4 1.59 0.19 0.20
query5 0.61 0.60 0.60
query6 1.22 0.70 0.72
query7 0.03 0.01 0.02
query8 0.04 0.03 0.04
query9 0.57 0.52 0.52
query10 0.55 0.59 0.57
query11 0.15 0.11 0.11
query12 0.14 0.11 0.12
query13 0.61 0.60 0.59
query14 2.79 2.83 2.83
query15 0.94 0.85 0.85
query16 0.40 0.39 0.38
query17 1.05 1.01 1.04
query18 0.22 0.19 0.19
query19 1.87 1.99 1.85
query20 0.01 0.02 0.01
query21 15.39 0.93 0.54
query22 0.76 1.11 0.78
query23 14.88 1.38 0.64
query24 6.65 1.98 0.94
query25 0.48 0.15 0.16
query26 0.61 0.16 0.14
query27 0.06 0.05 0.05
query28 9.99 0.88 0.44
query29 12.55 3.94 3.31
query30 0.25 0.08 0.06
query31 2.83 0.61 0.38
query32 3.22 0.56 0.47
query33 3.07 3.08 3.10
query34 15.78 5.10 4.52
query35 4.55 4.56 4.49
query36 0.66 0.50 0.49
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.16 0.13 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 105.39 s
Total hot run time: 31.67 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2774679394
<details>
<summary>TPC-H: <b>Total hot run time: 34108 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f02a3f0f717bfbd6f489653badc6514305e04b9, data reload: false
------ Round 1 ----------------------------------
q1 26195 5078 5041 5041
q2 2069 292 180 180
q3 10384 1226 714 714
q4 10223 1006 532 532
q5 7548 2352 2327 2327
q6 193 160 130 130
q7 911 739 597 597
q8 9329 1321 1054 1054
q9 6965 5075 5155 5075
q10 6811 2326 1889 1889
q11 475 284 265 265
q12 358 360 219 219
q13 17768 3686 3108 3108
q14 225 225 211 211
q15 535 510 488 488
q16 628 623 579 579
q17 587 859 378 378
q18 7580 7233 7056 7056
q19 1221 960 564 564
q20 338 348 239 239
q21 4110 3411 2500 2500
q22 1058 1003 962 962
Total cold run time: 115511 ms
Total hot run time: 34108 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5178 5162 5129 5129
q2 240 325 229 229
q3 2171 2686 2286 2286
q4 1425 1869 1414 1414
q5 4515 4488 4436 4436
q6 215 166 122 122
q7 1944 1899 1753 1753
q8 2579 2528 2467 2467
q9 7126 7131 7158 7131
q10 2974 3174 2727 2727
q11 576 500 497 497
q12 682 748 605 605
q13 3516 3850 3346 3346
q14 271 300 273 273
q15 526 467 482 467
q16 666 706 673 673
q17 1184 1609 1355 1355
q18 7801 7395 7510 7395
q19 798 791 940 791
q20 1992 2016 1823 1823
q21 5197 4728 4678 4678
q22 1015 1011 979 979
Total cold run time: 52591 ms
Total hot run time: 50576 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2701375457
<details>
<summary>TPC-DS: <b>Total hot run time: 191488 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4bf31488c6c90a028a6d62a484c8bc8a1f2b5539, data reload: false
query1 1382 1052 1023 1023
query2 6297 1877 1878 1877
query3 11022 4480 4374 4374
query4 54490 26298 23107 23107
query5 4915 511 475 475
query6 326 193 183 183
query7 4925 487 288 288
query8 303 226 224 224
query9 5385 2541 2553 2541
query10 424 317 266 266
query11 15054 15037 15013 15013
query12 155 107 105 105
query13 1023 499 389 389
query14 10874 6783 6668 6668
query15 203 202 186 186
query16 7115 697 523 523
query17 1107 741 594 594
query18 1620 419 331 331
query19 217 204 173 173
query20 136 127 126 126
query21 214 124 115 115
query22 4542 4263 4297 4263
query23 34190 33382 33277 33277
query24 5687 2416 2463 2416
query25 463 477 408 408
query26 690 280 150 150
query27 1823 518 335 335
query28 2814 2482 2450 2450
query29 541 556 435 435
query30 272 218 192 192
query31 894 875 803 803
query32 71 62 60 60
query33 460 359 305 305
query34 770 874 516 516
query35 790 853 792 792
query36 947 1008 898 898
query37 117 97 86 86
query38 4199 4351 4066 4066
query39 1483 1466 1427 1427
query40 214 113 103 103
query41 54 49 51 49
query42 135 110 115 110
query43 482 522 484 484
query44 1287 819 803 803
query45 186 174 175 174
query46 869 1040 643 643
query47 1821 1861 1792 1792
query48 404 418 310 310
query49 680 505 431 431
query50 726 770 422 422
query51 4256 4313 4223 4223
query52 116 103 98 98
query53 235 271 197 197
query54 482 494 428 428
query55 85 76 84 76
query56 287 290 263 263
query57 1145 1168 1109 1109
query58 243 245 248 245
query59 2838 2901 2575 2575
query60 284 279 278 278
query61 126 126 131 126
query62 779 777 705 705
query63 235 195 191 191
query64 1643 1021 685 685
query65 4527 4449 4268 4268
query66 699 445 294 294
query67 15723 15736 15277 15277
query68 7815 886 500 500
query69 543 302 263 263
query70 1197 1101 1130 1101
query71 474 299 278 278
query72 5711 3550 3662 3550
query73 1486 722 420 420
query74 9019 9194 8896 8896
query75 3687 3137 2709 2709
query76 4174 1187 744 744
query77 615 345 286 286
query78 10016 10149 9338 9338
query79 2651 825 594 594
query80 731 583 464 464
query81 486 254 227 227
query82 660 122 88 88
query83 296 167 152 152
query84 289 95 74 74
query85 825 353 309 309
query86 377 301 296 296
query87 4371 4470 4385 4385
query88 3466 2199 2158 2158
query89 428 316 293 293
query90 1938 193 192 192
query91 140 141 110 110
query92 73 70 55 55
query93 1327 1056 587 587
query94 688 392 308 308
query95 349 269 257 257
query96 494 549 265 265
query97 3348 3389 3310 3310
query98 235 208 195 195
query99 1453 1375 1265 1265
Total cold run time: 297863 ms
Total hot run time: 191488 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2700289234
<details>
<summary>ClickBench: <b>Total hot run time: 31.28 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d6b88a7861bafa88a7a6b898d05d0258359e7037, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.03
query3 0.24 0.07 0.07
query4 1.62 0.10 0.10
query5 0.55 0.55 0.54
query6 1.20 0.72 0.72
query7 0.03 0.03 0.02
query8 0.04 0.03 0.03
query9 0.58 0.54 0.52
query10 0.58 0.57 0.57
query11 0.15 0.11 0.11
query12 0.14 0.11 0.12
query13 0.61 0.61 0.59
query14 2.70 2.82 2.80
query15 0.94 0.85 0.85
query16 0.40 0.39 0.39
query17 1.05 1.02 1.04
query18 0.22 0.19 0.20
query19 1.89 1.81 2.03
query20 0.02 0.02 0.01
query21 15.41 0.87 0.55
query22 0.76 1.26 0.78
query23 14.73 1.37 0.65
query24 6.74 1.99 1.03
query25 0.48 0.35 0.09
query26 0.58 0.16 0.13
query27 0.04 0.05 0.06
query28 9.71 0.89 0.42
query29 12.56 3.93 3.29
query30 0.26 0.08 0.06
query31 2.83 0.60 0.39
query32 3.23 0.54 0.46
query33 2.98 3.02 3.03
query34 15.90 5.21 4.50
query35 4.53 4.59 4.56
query36 0.67 0.50 0.48
query37 0.09 0.06 0.06
query38 0.04 0.03 0.04
query39 0.03 0.02 0.03
query40 0.17 0.14 0.13
query41 0.08 0.02 0.03
query42 0.04 0.03 0.02
query43 0.03 0.03 0.04
Total cold run time: 104.96 s
Total hot run time: 31.28 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42932:
URL: https://github.com/apache/doris/pull/42932#issuecomment-2791671463
# BE UT Coverage Report
Increment line coverage `70.60% (778/1102)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49520d7e9cdd2eda186cfa5815b3db442739314f_49520d7e9cdd2eda186cfa5815b3db442739314f/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49520d7e9cdd2eda186cfa5815b3db442739314f_49520d7e9cdd2eda186cfa5815b3db442739314f/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.45% (14136/26952) |
| Line Coverage | 41.25% (122133/296075) |
| Region Coverage | 39.99% (62170/155473) |
| Branch Coverage | 34.67% (31162/89890) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Refactoring Phrase Query Implementation Algorithms with Inverted Index Optimization [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on code in PR #42932:
URL: https://github.com/apache/doris/pull/42932#discussion_r2011386784
##########
be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query/exact_phrase_matcher.cpp:
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/inverted_index/query/phrase_query/exact_phrase_matcher.h"
+
+namespace doris::segment_v2::inverted_index {
+
+ExactPhraseMatcher::ExactPhraseMatcher(std::vector<PostingsAndPosition> postings)
Review Comment:
Use code below to avoid params copy when call ExactPhraseMatcher constructor
`
class ExactPhraseMatcher {
public:
ExactPhraseMatcher(const std::vector<PostingsAndPosition>& postings)
: _postings(postings) {}
ExactPhraseMatcher(std::vector<PostingsAndPosition>&& postings)
: _postings(std::move(postings)) {}
};
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org