You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "csun5285 (via GitHub)" <gi...@apache.org> on 2025/05/09 06:17:30 UTC

[PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]

csun5285 opened a new pull request, #50748:
URL: https://github.com/apache/doris/pull/50748

   ### What problem does this PR solve?
   
   Support the IS NOT NULL condition in expr via inverted indexes
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [ ] Regression test
       - [ ] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
   - Behavior changed:
       - [ ] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [ ] No.
       - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]

Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #50748:
URL: https://github.com/apache/doris/pull/50748#discussion_r2083790663


##########
regression-test/suites/variant_p0/test_variant_is_null_expr.groovy:
##########
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+suite("test_variant_is_null_expr", "p0, nonConcurrent") {
+    // define a sql table
+    def testTable = "test_variant_is_null_expr"
+
+    sql """ DROP TABLE IF EXISTS ${testTable} """ 
+    sql """
+        CREATE TABLE ${testTable} (
+          `k` int(11) NULL COMMENT "",
+          `v` variant NULL COMMENT "",
+          INDEX idx_a (v) USING INVERTED
+          ) ENGINE=OLAP
+          DUPLICATE KEY(`k`)
+          COMMENT "OLAP"
+          DISTRIBUTED BY HASH(`k`) BUCKETS 1
+          PROPERTIES (
+          "replication_allocation" = "tag.location.default: 1"
+        );
+      """
+
+    sql """
+        INSERT INTO ${testTable} VALUES (1, '{"int1" : 1, "string1" : "aa"}'), (2, '{"int2" : 2, "string2" : "bb"}'), (3, '{"int3" : 3, "string3" : "cc"}');
+    """
+
+
+    def queryAndCheck = { String sqlQuery, int expectedFilteredRows = -1, boolean checkFilterUsed = true ->
+      def checkpoints_name = "segment_iterator.inverted_index.filtered_rows"
+      try {
+          GetDebugPoint().enableDebugPointForAllBEs("segment_iterator.apply_inverted_index")
+          GetDebugPoint().enableDebugPointForAllBEs(checkpoints_name, [filtered_rows: expectedFilteredRows])
+          sql "set experimental_enable_parallel_scan = false"
+          sql " set inverted_index_skip_threshold = 0 "
+          sql "sync"
+          sql "${sqlQuery}"
+      } finally {
+          GetDebugPoint().disableDebugPointForAllBEs(checkpoints_name)
+          GetDebugPoint().disableDebugPointForAllBEs("segment_iterator.apply_inverted_index")
+      }
+    }
+    
+    queryAndCheck (" select * from ${testTable} where v['int1'] is not null; ", 2)
+    queryAndCheck (" select * from ${testTable} where v['int1'] is null; ", 1)
+    queryAndCheck (" select * from ${testTable} where v['string1'] is not null; ", 2)
+    queryAndCheck (" select * from ${testTable} where v['string1'] is null; ", 1)
+    queryAndCheck (" select * from ${testTable} where v['int1'] is not null or v['string2'] is not null; ", 1)
+    queryAndCheck (" select * from ${testTable} where v['int1'] is not null or v['string2'] is null; ", 1)

Review Comment:
   add cast case



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865797233

   # BE Regression && UT Coverage Report
   Increment line coverage `92.98% (53/57)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/50748_44c0d0afe281b329a4ed6568ff34af0c8a1be0fd_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/50748_44c0d0afe281b329a4ed6568ff34af0c8a1be0fd_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 79.39% (20822/26228) |
   | Line Coverage     | 72.58% (213884/294693) |
   | Region Coverage   | 70.78% (126582/178827) |
   | Branch Coverage   | 64.40% (65218/101278) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865268633

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865592121

   # BE UT Coverage Report
   Increment line coverage `35.71% (20/56)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/44c0d0afe281b329a4ed6568ff34af0c8a1be0fd_44c0d0afe281b329a4ed6568ff34af0c8a1be0fd/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/44c0d0afe281b329a4ed6568ff34af0c8a1be0fd_44c0d0afe281b329a4ed6568ff34af0c8a1be0fd/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 55.68% (14872/26709) |
   | Line Coverage     | 44.48% (131173/294929) |
   | Region Coverage   | 43.38% (66102/152366) |
   | Branch Coverage   | 38.05% (33738/88676) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865388479

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 28.71 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 44c0d0afe281b329a4ed6568ff34af0c8a1be0fd, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.12	0.10	0.11
   query3	0.26	0.18	0.19
   query4	1.60	0.20	0.18
   query5	0.59	0.58	0.62
   query6	1.16	0.72	0.74
   query7	0.02	0.02	0.02
   query8	0.04	0.03	0.04
   query9	0.56	0.52	0.50
   query10	0.56	0.56	0.55
   query11	0.14	0.12	0.11
   query12	0.15	0.12	0.11
   query13	0.61	0.59	0.60
   query14	0.78	0.79	0.80
   query15	0.87	0.85	0.83
   query16	0.39	0.39	0.40
   query17	1.05	1.00	1.08
   query18	0.21	0.19	0.20
   query19	1.84	1.78	1.79
   query20	0.01	0.01	0.02
   query21	15.40	0.92	0.59
   query22	0.77	1.17	0.64
   query23	14.97	1.39	0.60
   query24	7.43	1.13	0.36
   query25	0.28	0.12	0.18
   query26	0.68	0.17	0.15
   query27	0.05	0.05	0.05
   query28	8.71	0.91	0.44
   query29	12.53	4.01	3.32
   query30	0.25	0.09	0.06
   query31	2.82	0.59	0.38
   query32	3.24	0.54	0.46
   query33	3.04	3.10	3.12
   query34	15.72	5.11	4.49
   query35	4.48	4.58	4.45
   query36	0.66	0.49	0.49
   query37	0.08	0.06	0.06
   query38	0.06	0.04	0.04
   query39	0.03	0.03	0.02
   query40	0.16	0.14	0.13
   query41	0.08	0.03	0.02
   query42	0.03	0.03	0.02
   query43	0.03	0.03	0.03
   Total cold run time: 102.5 s
   Total hot run time: 28.71 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865374623

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 191505 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 44c0d0afe281b329a4ed6568ff34af0c8a1be0fd, data reload: false
   
   query1	1406	1070	1071	1070
   query2	6257	1734	1758	1734
   query3	10988	4421	4315	4315
   query4	53437	25309	23325	23325
   query5	4994	508	456	456
   query6	349	213	196	196
   query7	4865	504	286	286
   query8	330	257	241	241
   query9	5646	2556	2541	2541
   query10	413	317	249	249
   query11	15085	15070	14797	14797
   query12	160	108	101	101
   query13	1026	497	387	387
   query14	10085	6226	6348	6226
   query15	198	195	173	173
   query16	7121	634	519	519
   query17	1073	727	562	562
   query18	1513	414	297	297
   query19	198	197	161	161
   query20	124	124	117	117
   query21	204	131	109	109
   query22	4365	4487	4306	4306
   query23	33877	33237	33117	33117
   query24	6629	2389	2443	2389
   query25	505	485	436	436
   query26	677	286	162	162
   query27	2138	514	338	338
   query28	3087	2146	2141	2141
   query29	604	597	454	454
   query30	270	216	190	190
   query31	882	872	820	820
   query32	78	67	63	63
   query33	456	413	310	310
   query34	784	859	524	524
   query35	789	834	766	766
   query36	934	1002	929	929
   query37	115	107	75	75
   query38	4123	4269	4192	4192
   query39	1545	1443	1487	1443
   query40	216	121	105	105
   query41	56	55	49	49
   query42	118	103	113	103
   query43	496	501	482	482
   query44	1295	810	792	792
   query45	187	174	165	165
   query46	831	1029	624	624
   query47	1851	1856	1849	1849
   query48	376	421	302	302
   query49	699	540	421	421
   query50	682	692	404	404
   query51	4201	4158	4104	4104
   query52	108	105	101	101
   query53	233	271	195	195
   query54	575	587	501	501
   query55	83	84	84	84
   query56	303	290	324	290
   query57	1151	1188	1096	1096
   query58	258	257	240	240
   query59	2673	2767	2625	2625
   query60	348	352	321	321
   query61	143	154	124	124
   query62	734	747	683	683
   query63	222	190	188	188
   query64	1450	1031	719	719
   query65	4410	4261	4150	4150
   query66	724	404	313	313
   query67	15620	15469	15373	15373
   query68	7155	886	497	497
   query69	524	293	254	254
   query70	1213	1065	1098	1065
   query71	498	315	289	289
   query72	5562	4700	4895	4700
   query73	1469	667	344	344
   query74	9163	9074	9054	9054
   query75	3814	3256	2696	2696
   query76	4195	1201	750	750
   query77	635	365	275	275
   query78	10138	9982	9170	9170
   query79	2235	794	559	559
   query80	589	515	434	434
   query81	486	254	215	215
   query82	464	128	95	95
   query83	249	239	236	236
   query84	294	99	95	95
   query85	776	351	323	323
   query86	389	309	301	301
   query87	4429	4412	4369	4369
   query88	3778	2240	2211	2211
   query89	433	317	278	278
   query90	1805	211	216	211
   query91	144	137	111	111
   query92	79	58	54	54
   query93	1781	954	578	578
   query94	669	406	298	298
   query95	366	293	287	287
   query96	492	568	278	278
   query97	3123	3273	3109	3109
   query98	229	202	197	197
   query99	1583	1395	1252	1252
   Total cold run time: 296132 ms
   Total hot run time: 191505 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]

Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865268810

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](inverted index) Support the IS NOT NULL condition in expr via inverted indexes [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50748:
URL: https://github.com/apache/doris/pull/50748#issuecomment-2865346312

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 33725 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 44c0d0afe281b329a4ed6568ff34af0c8a1be0fd, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	26056	5556	4989	4989
   q2	2094	289	187	187
   q3	10378	1217	687	687
   q4	10229	979	511	511
   q5	7524	2316	2317	2316
   q6	178	163	131	131
   q7	919	745	604	604
   q8	9323	1183	1043	1043
   q9	6753	5092	5075	5075
   q10	6851	2273	1884	1884
   q11	472	294	256	256
   q12	340	363	212	212
   q13	17779	3699	3092	3092
   q14	225	230	213	213
   q15	535	484	492	484
   q16	419	423	367	367
   q17	588	842	360	360
   q18	7564	7078	7208	7078
   q19	1377	945	582	582
   q20	324	331	229	229
   q21	4167	3342	2460	2460
   q22	1017	1020	965	965
   Total cold run time: 115112 ms
   Total hot run time: 33725 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	5114	5113	5050	5050
   q2	242	336	234	234
   q3	2282	2704	2396	2396
   q4	1429	1857	1409	1409
   q5	4421	4391	4368	4368
   q6	215	170	124	124
   q7	1985	1888	1753	1753
   q8	2577	2459	2533	2459
   q9	7157	7159	6893	6893
   q10	3044	3164	2729	2729
   q11	562	518	512	512
   q12	688	773	645	645
   q13	3486	3937	3250	3250
   q14	268	314	279	279
   q15	537	499	500	499
   q16	431	468	431	431
   q17	1128	1528	1403	1403
   q18	7753	7541	7393	7393
   q19	823	823	891	823
   q20	1979	2011	1842	1842
   q21	5007	4842	4653	4653
   q22	1071	1099	1041	1041
   Total cold run time: 52199 ms
   Total hot run time: 50186 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org