You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "pzhdfy (via GitHub)" <gi...@apache.org> on 2025/11/11 07:30:38 UTC
[PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
pzhdfy opened a new pull request, #57902:
URL: https://github.com/apache/doris/pull/57902
### What problem does this PR solve?
Problem Summary:
all reproduce detail can be found in TEST(BitmapValueTest, set_extra)
case1:
BitmapValue::empty() and BitmapValue::minimum() is not correct for a BitmapValue with empty set
case2:
BitmapValue xor operator can make a SET type with more than 32 element
so deserialize will crash when CHECK(count <= SET_TYPE_THRESHOLD)
<img width="516" height="251" alt="image" src="https://github.com/user-attachments/assets/1e7e4145-b6ca-4763-8e2d-fbc7b67d8625" />
also can be reproduced with the following sql
```
select BITMAP_UNION(x)
from
(
select group_bitmap_xor(b) as x
from (select to_bitmap(number) as b from numbers("number" = "64")) t
union all
select group_bitmap_xor(b) as x
from (select to_bitmap(number) as b from numbers("number" = "64")) t2
) t3;
```
case3:
BitmapValue xor operator does not clear set, when set covert to bitmap.
<img width="276" height="268" alt="image" src="https://github.com/user-attachments/assets/e1233931-48a3-432c-b271-ab0227cc7ea8" />
then _convert_to_smaller_type will back to set, when bitmap card = 1, and the final set is not correct because
_set has old item
<img width="669" height="567" alt="image" src="https://github.com/user-attachments/assets/9183bd77-6bf4-4eae-ac67-735905dffd94" />
also can be reproduced with the following sql
```
select
bitmap_to_array(
bitmap_xor(
bitmap_and(
bitmap_from_array([1,2]),
bitmap_from_array([2,3])
),
bitmap_and(
bitmap_from_array(
[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34]
),
bitmap_from_array(
[2,3,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134]
)
)
)
);
```
case4:
BitmapValue or operator not clear set, when set covert to bitmap
<img width="356" height="212" alt="image" src="https://github.com/user-attachments/assets/4e040abf-a153-4ac9-bdc9-a6f6c381a510" />
then this BitmapValue call and operator , may get incorrect result
<img width="451" height="271" alt="image" src="https://github.com/user-attachments/assets/4b356ea2-8e41-4307-a18d-190e6d589232" />
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #57902:
URL: https://github.com/apache/doris/pull/57902#issuecomment-3695891622
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #57902: [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result)
URL: https://github.com/apache/doris/pull/57902
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #57902:
URL: https://github.com/apache/doris/pull/57902#issuecomment-3696634576
# BE Regression && UT Coverage Report
Increment line coverage `82.76% (24/29)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/57902_9b0b30f1e2038a77b6c702840ae620e442d514f7_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/57902_9b0b30f1e2038a77b6c702840ae620e442d514f7_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.23% (25066/34703) |
| Line Coverage | 58.97% (263331/446540) |
| Region Coverage | 53.91% (219031/406319) |
| Branch Coverage | 55.43% (93986/169549) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #57902:
URL: https://github.com/apache/doris/pull/57902#issuecomment-3695968968
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #57902:
URL: https://github.com/apache/doris/pull/57902#issuecomment-3696285987
<details>
<summary>TPC-DS: <b>Total hot run time: 175714 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9b0b30f1e2038a77b6c702840ae620e442d514f7, data reload: false
query5 4421 596 431 431
query6 331 220 207 207
query7 4222 461 276 276
query8 318 245 246 245
query9 8772 2615 2671 2615
query10 517 363 334 334
query11 15274 15130 14772 14772
query12 178 121 116 116
query13 1278 516 391 391
query14 5970 2939 2716 2716
query14_1 2588 2617 2630 2617
query15 204 191 175 175
query16 1005 488 458 458
query17 1110 699 618 618
query18 2444 442 342 342
query19 225 225 193 193
query20 123 116 118 116
query21 217 138 122 122
query22 4010 4149 4216 4149
query23 16011 15521 15343 15343
query23_1 15491 15498 15519 15498
query24 7382 1600 1234 1234
query24_1 1250 1208 1256 1208
query25 559 472 436 436
query26 1227 275 164 164
query27 2753 462 301 301
query28 4558 2209 2197 2197
query29 852 561 450 450
query30 319 242 214 214
query31 790 631 560 560
query32 87 78 72 72
query33 561 338 298 298
query34 950 868 525 525
query35 756 789 711 711
query36 877 863 821 821
query37 136 91 76 76
query38 2659 2768 2648 2648
query39 755 740 721 721
query39_1 720 710 704 704
query40 221 128 113 113
query41 64 63 64 63
query42 107 112 100 100
query43 474 443 425 425
query44 1335 764 748 748
query45 188 183 182 182
query46 863 968 635 635
query47 1365 1399 1287 1287
query48 319 322 251 251
query49 607 413 324 324
query50 641 277 205 205
query51 3770 3722 3734 3722
query52 104 105 96 96
query53 289 325 269 269
query54 277 244 241 241
query55 83 79 73 73
query56 324 287 277 277
query57 980 1036 955 955
query58 261 248 241 241
query59 1981 2045 2212 2045
query60 310 314 285 285
query61 158 159 160 159
query62 402 395 323 323
query63 296 261 264 261
query64 5100 1294 1007 1007
query65 3832 3701 3708 3701
query66 1435 427 313 313
query67 15608 16076 15883 15883
query68 6890 1009 735 735
query69 508 347 308 308
query70 1044 943 944 943
query71 378 316 274 274
query72 6181 4717 4783 4717
query73 666 586 315 315
query74 8869 8850 8613 8613
query75 2865 2888 2539 2539
query76 3855 1063 658 658
query77 512 377 280 280
query78 9703 9975 9185 9185
query79 1257 867 607 607
query80 610 591 483 483
query81 510 276 234 234
query82 210 146 112 112
query83 271 254 248 248
query84 252 111 102 102
query85 879 499 453 453
query86 385 322 306 306
query87 2854 2825 2721 2721
query88 3187 2264 2262 2262
query89 378 336 330 330
query90 2179 153 157 153
query91 189 186 163 163
query92 81 66 61 61
query93 1188 901 557 557
query94 569 312 294 294
query95 577 377 295 295
query96 599 467 210 210
query97 2343 2390 2303 2303
query98 219 206 203 203
query99 617 577 539 539
Total cold run time: 253191 ms
Total hot run time: 175714 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #57902:
URL: https://github.com/apache/doris/pull/57902#discussion_r2654918697
##########
be/src/util/bitmap_value.h:
##########
@@ -2038,8 +2046,13 @@ class BitmapValue {
case BITMAP:
return _bitmap->minimum();
case SET:
+ if (_set.empty()) {
Review Comment:
I think we don't need to modify this; min_or_max will handle empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #57902:
URL: https://github.com/apache/doris/pull/57902#issuecomment-3516310103
great work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #57902:
URL: https://github.com/apache/doris/pull/57902#issuecomment-3695968856
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #57902:
URL: https://github.com/apache/doris/pull/57902#discussion_r2654914638
##########
be/src/util/bitmap_value.h:
##########
@@ -1502,6 +1505,7 @@ class BitmapValue {
break;
case BITMAP:
_prepare_bitmap_for_write();
+ _set.clear();
Review Comment:
does we need check set is empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #57902:
URL: https://github.com/apache/doris/pull/57902#issuecomment-3696477513
# BE UT Coverage Report
Increment line coverage `82.76% (24/29)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9b0b30f1e2038a77b6c702840ae620e442d514f7_9b0b30f1e2038a77b6c702840ae620e442d514f7/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9b0b30f1e2038a77b6c702840ae620e442d514f7_9b0b30f1e2038a77b6c702840ae620e442d514f7/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.39% (18950/35496) |
| Line Coverage | 39.26% (175795/447723) |
| Region Coverage | 33.82% (135989/402152) |
| Branch Coverage | 34.76% (58734/168985) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #57902:
URL: https://github.com/apache/doris/pull/57902#issuecomment-3515343710
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #57902:
URL: https://github.com/apache/doris/pull/57902#issuecomment-3696265500
<details>
<summary>TPC-H: <b>Total hot run time: 34381 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9b0b30f1e2038a77b6c702840ae620e442d514f7, data reload: false
------ Round 1 ----------------------------------
q1 17617 4257 4077 4077
q2 2022 359 242 242
q3 10182 1268 758 758
q4 10229 859 327 327
q5 7681 2208 1932 1932
q6 220 178 142 142
q7 939 797 678 678
q8 9272 1436 1170 1170
q9 7013 5153 5124 5124
q10 6834 1786 1432 1432
q11 515 313 297 297
q12 756 725 607 607
q13 17785 3849 3039 3039
q14 302 290 276 276
q15 598 510 506 506
q16 701 681 650 650
q17 713 798 665 665
q18 7982 7286 7230 7230
q19 1111 1120 643 643
q20 402 393 247 247
q21 4205 3911 3352 3352
q22 1064 1034 987 987
Total cold run time: 108143 ms
Total hot run time: 34381 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4120 4057 4047 4047
q2 347 402 322 322
q3 2131 2570 2286 2286
q4 1379 1754 1321 1321
q5 4127 4293 4668 4293
q6 223 174 128 128
q7 2073 1947 1757 1757
q8 2484 2373 2363 2363
q9 7200 7187 7088 7088
q10 2513 2704 2270 2270
q11 582 493 462 462
q12 732 722 665 665
q13 3820 4055 3399 3399
q14 299 310 276 276
q15 532 505 501 501
q16 698 710 701 701
q17 1200 1499 1406 1406
q18 7695 7704 7792 7704
q19 943 907 903 903
q20 1999 2095 1898 1898
q21 4451 4279 4072 4072
q22 1084 1041 1018 1018
Total cold run time: 50632 ms
Total hot run time: 48880 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](bitmap) fix several issues when enable_set_in_bitmap_value (may crash or get incrorrect result) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #57902:
URL: https://github.com/apache/doris/pull/57902#issuecomment-3696311008
<details>
<summary>ClickBench: <b>Total hot run time: 27.39 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9b0b30f1e2038a77b6c702840ae620e442d514f7, data reload: false
query1 0.05 0.05 0.05
query2 0.10 0.05 0.06
query3 0.26 0.10 0.09
query4 1.61 0.11 0.11
query5 0.27 0.26 0.27
query6 1.15 0.68 0.65
query7 0.03 0.03 0.03
query8 0.05 0.04 0.04
query9 0.55 0.50 0.50
query10 0.55 0.56 0.53
query11 0.16 0.11 0.11
query12 0.16 0.13 0.13
query13 0.61 0.60 0.60
query14 0.99 0.98 0.98
query15 0.80 0.78 0.79
query16 0.39 0.40 0.42
query17 1.08 1.07 1.05
query18 0.22 0.21 0.21
query19 1.94 1.87 1.80
query20 0.02 0.01 0.02
query21 15.71 0.30 0.15
query22 5.38 0.05 0.05
query23 16.07 0.28 0.10
query24 1.02 0.65 0.64
query25 0.09 0.06 0.07
query26 0.14 0.13 0.14
query27 0.08 0.06 0.06
query28 4.38 1.05 0.88
query29 12.64 3.89 3.17
query30 0.29 0.14 0.12
query31 2.82 0.64 0.38
query32 3.23 0.56 0.46
query33 3.03 3.02 3.11
query34 16.61 5.15 4.49
query35 4.52 4.48 4.49
query36 0.66 0.49 0.48
query37 0.10 0.07 0.06
query38 0.07 0.04 0.03
query39 0.04 0.04 0.03
query40 0.17 0.14 0.13
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 98.21 s
Total hot run time: 27.39 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org