You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "csun5285 (via GitHub)" <gi...@apache.org> on 2025/05/09 06:17:30 UTC
[PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]
csun5285 opened a new pull request, #50748:
URL: https://github.com/apache/doris/pull/50748
### What problem does this PR solve?
Support the IS NOT NULL condition in expr via inverted indexes
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #50748:
URL: https://github.com/apache/doris/pull/50748#discussion_r2083790663
##########
regression-test/suites/variant_p0/test_variant_is_null_expr.groovy:
##########
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+suite("test_variant_is_null_expr", "p0, nonConcurrent") {
+ // define a sql table
+ def testTable = "test_variant_is_null_expr"
+
+ sql """ DROP TABLE IF EXISTS ${testTable} """
+ sql """
+ CREATE TABLE ${testTable} (
+ `k` int(11) NULL COMMENT "",
+ `v` variant NULL COMMENT "",
+ INDEX idx_a (v) USING INVERTED
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`k`)
+ COMMENT "OLAP"
+ DISTRIBUTED BY HASH(`k`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ INSERT INTO ${testTable} VALUES (1, '{"int1" : 1, "string1" : "aa"}'), (2, '{"int2" : 2, "string2" : "bb"}'), (3, '{"int3" : 3, "string3" : "cc"}');
+ """
+
+
+ def queryAndCheck = { String sqlQuery, int expectedFilteredRows = -1, boolean checkFilterUsed = true ->
+ def checkpoints_name = "segment_iterator.inverted_index.filtered_rows"
+ try {
+ GetDebugPoint().enableDebugPointForAllBEs("segment_iterator.apply_inverted_index")
+ GetDebugPoint().enableDebugPointForAllBEs(checkpoints_name, [filtered_rows: expectedFilteredRows])
+ sql "set experimental_enable_parallel_scan = false"
+ sql " set inverted_index_skip_threshold = 0 "
+ sql "sync"
+ sql "${sqlQuery}"
+ } finally {
+ GetDebugPoint().disableDebugPointForAllBEs(checkpoints_name)
+ GetDebugPoint().disableDebugPointForAllBEs("segment_iterator.apply_inverted_index")
+ }
+ }
+
+ queryAndCheck (" select * from ${testTable} where v['int1'] is not null; ", 2)
+ queryAndCheck (" select * from ${testTable} where v['int1'] is null; ", 1)
+ queryAndCheck (" select * from ${testTable} where v['string1'] is not null; ", 2)
+ queryAndCheck (" select * from ${testTable} where v['string1'] is null; ", 1)
+ queryAndCheck (" select * from ${testTable} where v['int1'] is not null or v['string2'] is not null; ", 1)
+ queryAndCheck (" select * from ${testTable} where v['int1'] is not null or v['string2'] is null; ", 1)
Review Comment:
add cast case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865797233
# BE Regression && UT Coverage Report
Increment line coverage `92.98% (53/57)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/50748_44c0d0afe281b329a4ed6568ff34af0c8a1be0fd_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/50748_44c0d0afe281b329a4ed6568ff34af0c8a1be0fd_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.39% (20822/26228) |
| Line Coverage | 72.58% (213884/294693) |
| Region Coverage | 70.78% (126582/178827) |
| Branch Coverage | 64.40% (65218/101278) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865268633
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865592121
# BE UT Coverage Report
Increment line coverage `35.71% (20/56)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/44c0d0afe281b329a4ed6568ff34af0c8a1be0fd_44c0d0afe281b329a4ed6568ff34af0c8a1be0fd/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/44c0d0afe281b329a4ed6568ff34af0c8a1be0fd_44c0d0afe281b329a4ed6568ff34af0c8a1be0fd/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 55.68% (14872/26709) |
| Line Coverage | 44.48% (131173/294929) |
| Region Coverage | 43.38% (66102/152366) |
| Branch Coverage | 38.05% (33738/88676) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865388479
<details>
<summary>ClickBench: <b>Total hot run time: 28.71 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 44c0d0afe281b329a4ed6568ff34af0c8a1be0fd, data reload: false
query1 0.04 0.03 0.03
query2 0.12 0.10 0.11
query3 0.26 0.18 0.19
query4 1.60 0.20 0.18
query5 0.59 0.58 0.62
query6 1.16 0.72 0.74
query7 0.02 0.02 0.02
query8 0.04 0.03 0.04
query9 0.56 0.52 0.50
query10 0.56 0.56 0.55
query11 0.14 0.12 0.11
query12 0.15 0.12 0.11
query13 0.61 0.59 0.60
query14 0.78 0.79 0.80
query15 0.87 0.85 0.83
query16 0.39 0.39 0.40
query17 1.05 1.00 1.08
query18 0.21 0.19 0.20
query19 1.84 1.78 1.79
query20 0.01 0.01 0.02
query21 15.40 0.92 0.59
query22 0.77 1.17 0.64
query23 14.97 1.39 0.60
query24 7.43 1.13 0.36
query25 0.28 0.12 0.18
query26 0.68 0.17 0.15
query27 0.05 0.05 0.05
query28 8.71 0.91 0.44
query29 12.53 4.01 3.32
query30 0.25 0.09 0.06
query31 2.82 0.59 0.38
query32 3.24 0.54 0.46
query33 3.04 3.10 3.12
query34 15.72 5.11 4.49
query35 4.48 4.58 4.45
query36 0.66 0.49 0.49
query37 0.08 0.06 0.06
query38 0.06 0.04 0.04
query39 0.03 0.03 0.02
query40 0.16 0.14 0.13
query41 0.08 0.03 0.02
query42 0.03 0.03 0.02
query43 0.03 0.03 0.03
Total cold run time: 102.5 s
Total hot run time: 28.71 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865374623
<details>
<summary>TPC-DS: <b>Total hot run time: 191505 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 44c0d0afe281b329a4ed6568ff34af0c8a1be0fd, data reload: false
query1 1406 1070 1071 1070
query2 6257 1734 1758 1734
query3 10988 4421 4315 4315
query4 53437 25309 23325 23325
query5 4994 508 456 456
query6 349 213 196 196
query7 4865 504 286 286
query8 330 257 241 241
query9 5646 2556 2541 2541
query10 413 317 249 249
query11 15085 15070 14797 14797
query12 160 108 101 101
query13 1026 497 387 387
query14 10085 6226 6348 6226
query15 198 195 173 173
query16 7121 634 519 519
query17 1073 727 562 562
query18 1513 414 297 297
query19 198 197 161 161
query20 124 124 117 117
query21 204 131 109 109
query22 4365 4487 4306 4306
query23 33877 33237 33117 33117
query24 6629 2389 2443 2389
query25 505 485 436 436
query26 677 286 162 162
query27 2138 514 338 338
query28 3087 2146 2141 2141
query29 604 597 454 454
query30 270 216 190 190
query31 882 872 820 820
query32 78 67 63 63
query33 456 413 310 310
query34 784 859 524 524
query35 789 834 766 766
query36 934 1002 929 929
query37 115 107 75 75
query38 4123 4269 4192 4192
query39 1545 1443 1487 1443
query40 216 121 105 105
query41 56 55 49 49
query42 118 103 113 103
query43 496 501 482 482
query44 1295 810 792 792
query45 187 174 165 165
query46 831 1029 624 624
query47 1851 1856 1849 1849
query48 376 421 302 302
query49 699 540 421 421
query50 682 692 404 404
query51 4201 4158 4104 4104
query52 108 105 101 101
query53 233 271 195 195
query54 575 587 501 501
query55 83 84 84 84
query56 303 290 324 290
query57 1151 1188 1096 1096
query58 258 257 240 240
query59 2673 2767 2625 2625
query60 348 352 321 321
query61 143 154 124 124
query62 734 747 683 683
query63 222 190 188 188
query64 1450 1031 719 719
query65 4410 4261 4150 4150
query66 724 404 313 313
query67 15620 15469 15373 15373
query68 7155 886 497 497
query69 524 293 254 254
query70 1213 1065 1098 1065
query71 498 315 289 289
query72 5562 4700 4895 4700
query73 1469 667 344 344
query74 9163 9074 9054 9054
query75 3814 3256 2696 2696
query76 4195 1201 750 750
query77 635 365 275 275
query78 10138 9982 9170 9170
query79 2235 794 559 559
query80 589 515 434 434
query81 486 254 215 215
query82 464 128 95 95
query83 249 239 236 236
query84 294 99 95 95
query85 776 351 323 323
query86 389 309 301 301
query87 4429 4412 4369 4369
query88 3778 2240 2211 2211
query89 433 317 278 278
query90 1805 211 216 211
query91 144 137 111 111
query92 79 58 54 54
query93 1781 954 578 578
query94 669 406 298 298
query95 366 293 287 287
query96 492 568 278 278
query97 3123 3273 3109 3109
query98 229 202 197 197
query99 1583 1395 1252 1252
Total cold run time: 296132 ms
Total hot run time: 191505 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865268810
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865346312
<details>
<summary>TPC-H: <b>Total hot run time: 33725 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 44c0d0afe281b329a4ed6568ff34af0c8a1be0fd, data reload: false
------ Round 1 ----------------------------------
q1 26056 5556 4989 4989
q2 2094 289 187 187
q3 10378 1217 687 687
q4 10229 979 511 511
q5 7524 2316 2317 2316
q6 178 163 131 131
q7 919 745 604 604
q8 9323 1183 1043 1043
q9 6753 5092 5075 5075
q10 6851 2273 1884 1884
q11 472 294 256 256
q12 340 363 212 212
q13 17779 3699 3092 3092
q14 225 230 213 213
q15 535 484 492 484
q16 419 423 367 367
q17 588 842 360 360
q18 7564 7078 7208 7078
q19 1377 945 582 582
q20 324 331 229 229
q21 4167 3342 2460 2460
q22 1017 1020 965 965
Total cold run time: 115112 ms
Total hot run time: 33725 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5114 5113 5050 5050
q2 242 336 234 234
q3 2282 2704 2396 2396
q4 1429 1857 1409 1409
q5 4421 4391 4368 4368
q6 215 170 124 124
q7 1985 1888 1753 1753
q8 2577 2459 2533 2459
q9 7157 7159 6893 6893
q10 3044 3164 2729 2729
q11 562 518 512 512
q12 688 773 645 645
q13 3486 3937 3250 3250
q14 268 314 279 279
q15 537 499 500 499
q16 431 468 431 431
q17 1128 1528 1403 1403
q18 7753 7541 7393 7393
q19 823 823 891 823
q20 1979 2011 1842 1842
q21 5007 4842 4653 4653
q22 1071 1099 1041 1041
Total cold run time: 52199 ms
Total hot run time: 50186 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org