You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "rohitrs1983 (via GitHub)" <gi...@apache.org> on 2024/03/12 13:06:52 UTC
[PR] [bugfix]json_length() BE crash fix [doris]
rohitrs1983 opened a new pull request, #32145:
URL: https://github.com/apache/doris/pull/32145
## Proposed changes
Issue Number: close #32124
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-1994020618
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-1994085123
TeamCity be ut coverage result:
Function Coverage: 35.01% (8576/24499)
Line Coverage: 26.76% (69475/259623)
Region Coverage: 26.03% (36081/138609)
Branch Coverage: 22.98% (18421/80152)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3c62ebe28fffd4eb4f1d8d634be9700c01c0995d_3c62ebe28fffd4eb4f1d8d634be9700c01c0995d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-1991611672
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-1994100890
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 3c62ebe28fffd4eb4f1d8d634be9700c01c0995d with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.9 seconds inserted 10000000 Rows, about 719K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-2006858282
@rohitrs1983 I've picked it to branch-2.0 as #32484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "rohitrs1983 (via GitHub)" <gi...@apache.org>.
rohitrs1983 commented on code in PR #32145:
URL: https://github.com/apache/doris/pull/32145#discussion_r1522199340
##########
be/src/vec/functions/function_jsonb.cpp:
##########
@@ -1097,10 +1097,15 @@ struct JsonbLengthUtil {
}
}
auto jsonb_value = jsonb_data_column->get_data_at(i);
+ if (UNLIKELY(jsonb_value.size == 0)) {
+ null_map->get_data()[i] = 1;
+ res->insert_data(nullptr, 0);
+ continue;
Review Comment:
updated. added regression test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-1994095470
<details>
<summary>ClickBench: <b>Total hot run time: 29.33 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3c62ebe28fffd4eb4f1d8d634be9700c01c0995d, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.02
query3 0.23 0.06 0.06
query4 1.66 0.10 0.10
query5 0.52 0.51 0.52
query6 1.20 0.63 0.62
query7 0.02 0.01 0.02
query8 0.04 0.02 0.02
query9 0.55 0.50 0.50
query10 0.55 0.56 0.54
query11 0.12 0.09 0.10
query12 0.11 0.10 0.09
query13 0.57 0.56 0.57
query14 0.75 0.76 0.75
query15 0.79 0.77 0.77
query16 0.40 0.38 0.38
query17 0.95 0.99 0.94
query18 0.20 0.25 0.24
query19 1.71 1.80 1.64
query20 0.01 0.01 0.01
query21 15.41 0.58 0.56
query22 2.64 2.25 1.35
query23 17.24 0.81 0.70
query24 3.07 0.60 1.60
query25 0.10 0.08 0.13
query26 0.63 0.13 0.12
query27 0.05 0.04 0.03
query28 11.07 0.90 0.85
query29 12.55 3.17 3.10
query30 0.66 0.61 0.57
query31 2.81 0.34 0.34
query32 3.37 0.43 0.43
query33 2.74 2.80 2.81
query34 16.12 4.34 4.31
query35 4.37 4.34 4.36
query36 1.06 1.00 1.01
query37 0.07 0.06 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.02
query40 0.16 0.12 0.13
query41 0.08 0.02 0.02
query42 0.03 0.01 0.01
query43 0.03 0.02 0.02
Total cold run time: 104.8 s
Total hot run time: 29.33 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #32145:
URL: https://github.com/apache/doris/pull/32145
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "cambyzju (via GitHub)" <gi...@apache.org>.
cambyzju commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-1998869606
@rohitrs1983 could you also fix this problem on branch-2.0 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-1993664803
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-1994085826
<details>
<summary>TPC-DS: <b>Total hot run time: 186750 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c62ebe28fffd4eb4f1d8d634be9700c01c0995d, data reload: false
query1 923 356 336 336
query2 7501 2076 2034 2034
query3 6720 215 217 215
query4 31186 20729 20861 20729
query5 4423 417 396 396
query6 261 176 165 165
query7 4625 291 284 284
query8 228 168 158 158
query9 9888 2276 2281 2276
query10 440 241 239 239
query11 14594 14109 14156 14109
query12 133 88 84 84
query13 1631 414 430 414
query14 11708 10605 10251 10251
query15 260 212 201 201
query16 8080 263 244 244
query17 1939 554 512 512
query18 2073 286 282 282
query19 259 152 144 144
query20 88 91 79 79
query21 198 138 130 130
query22 4652 4555 4572 4555
query23 31783 30902 31137 30902
query24 10797 2826 2834 2826
query25 629 391 365 365
query26 1417 165 154 154
query27 3033 350 341 341
query28 8142 1891 1852 1852
query29 927 667 615 615
query30 346 149 151 149
query31 962 731 719 719
query32 98 61 55 55
query33 765 259 254 254
query34 1059 470 473 470
query35 832 600 588 588
query36 1016 875 859 859
query37 128 75 77 75
query38 3334 3184 3187 3184
query39 1393 1366 1375 1366
query40 202 110 110 110
query41 53 47 47 47
query42 103 92 97 92
query43 495 467 457 457
query44 1308 736 724 724
query45 271 247 256 247
query46 1110 688 682 682
query47 1669 1605 1616 1605
query48 434 351 350 350
query49 1141 342 343 342
query50 751 410 372 372
query51 6591 6580 6634 6580
query52 103 92 87 87
query53 343 275 276 275
query54 298 235 234 234
query55 83 80 82 80
query56 268 233 224 224
query57 1089 1025 1024 1024
query58 231 199 203 199
query59 2733 2589 2568 2568
query60 264 282 246 246
query61 110 108 107 107
query62 591 403 393 393
query63 296 266 274 266
query64 5632 3942 3754 3754
query65 3123 3018 2999 2999
query66 1448 334 320 320
query67 15173 14628 14485 14485
query68 7032 527 529 527
query69 643 378 374 374
query70 1272 1187 1155 1155
query71 528 290 267 267
query72 6889 2651 2471 2471
query73 728 319 313 313
query74 6847 6527 6598 6527
query75 9394 7966 7988 7966
query76 5012 927 999 927
query77 641 258 252 252
query78 10488 9683 9772 9683
query79 11315 519 511 511
query80 1757 392 368 368
query81 530 207 208 207
query82 866 203 205 203
query83 207 140 140 140
query84 281 74 75 74
query85 1504 321 302 302
query86 480 302 294 294
query87 3500 3298 3258 3258
query88 5185 2265 2246 2246
query89 524 358 361 358
query90 1949 172 170 170
query91 159 126 125 125
query92 55 45 46 45
query93 7570 513 493 493
query94 1231 177 172 172
query95 435 326 321 321
query96 603 271 264 264
query97 4436 4291 4297 4291
query98 227 220 213 213
query99 1274 782 753 753
Total cold run time: 317946 ms
Total hot run time: 186750 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-1994062509
<details>
<summary>TPC-H: <b>Total hot run time: 36970 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c62ebe28fffd4eb4f1d8d634be9700c01c0995d, data reload: false
------ Round 1 ----------------------------------
q1 18011 4336 4177 4177
q2 2213 159 146 146
q3 11162 939 951 939
q4 5299 703 729 703
q5 8016 2881 2898 2881
q6 188 125 128 125
q7 1188 850 817 817
q8 9650 2138 2097 2097
q9 7452 6521 6506 6506
q10 8131 2466 2496 2466
q11 421 224 226 224
q12 829 297 303 297
q13 17956 3013 2968 2968
q14 282 253 250 250
q15 497 451 450 450
q16 471 390 403 390
q17 941 533 559 533
q18 6664 5824 5776 5776
q19 1555 1491 1390 1390
q20 555 273 289 273
q21 7365 3250 3256 3250
q22 801 312 333 312
Total cold run time: 109647 ms
Total hot run time: 36970 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4141 4056 4099 4056
q2 318 222 220 220
q3 2971 2870 2820 2820
q4 1819 1534 1513 1513
q5 5211 5234 5253 5234
q6 196 114 124 114
q7 2243 1832 1870 1832
q8 3128 3249 3285 3249
q9 8601 8591 8586 8586
q10 6174 3669 3697 3669
q11 530 432 449 432
q12 722 542 592 542
q13 8995 2890 2842 2842
q14 262 246 257 246
q15 475 451 449 449
q16 449 417 410 410
q17 1706 1495 1452 1452
q18 7714 7261 7250 7250
q19 1636 1500 1561 1500
q20 1926 1719 1704 1704
q21 4853 4706 4665 4665
q22 531 476 455 455
Total cold run time: 64601 ms
Total hot run time: 53240 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32145:
URL: https://github.com/apache/doris/pull/32145#discussion_r1521497768
##########
be/src/vec/functions/function_jsonb.cpp:
##########
@@ -1097,10 +1097,15 @@ struct JsonbLengthUtil {
}
}
auto jsonb_value = jsonb_data_column->get_data_at(i);
+ if (UNLIKELY(jsonb_value.size == 0)) {
+ null_map->get_data()[i] = 1;
+ res->insert_data(nullptr, 0);
+ continue;
Review Comment:
please add the test case in the issue to regression test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix]json_length() BE crash fix [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32145:
URL: https://github.com/apache/doris/pull/32145#issuecomment-1993664767
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org