You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Chen768959 (via GitHub)" <gi...@apache.org> on 2025/11/13 13:46:21 UTC
[PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Chen768959 opened a new pull request, #58002:
URL: https://github.com/apache/doris/pull/58002
Fix memory alignment issues in read_column_from_arrow by using memcpy
This PR addresses potential memory alignment problems when reading data from Arrow arrays in the `read_column_from_arrow` method. Instead of directly casting potentially unaligned pointers, we now use `memcpy` to safely copy data into properly aligned memory, preventing misaligned address runtime errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3557167648
<details>
<summary>TPC-DS: <b>Total hot run time: 188375 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0a24ddd7b4ed3fe7f060f4c675e12b7655f3b084, data reload: false
query1 1036 410 427 410
query2 6570 1722 1694 1694
query3 6764 222 224 222
query4 26231 23331 23265 23265
query5 4399 621 491 491
query6 352 236 225 225
query7 4645 497 306 306
query8 330 249 244 244
query9 8711 2606 2577 2577
query10 511 338 316 316
query11 15731 14979 14849 14849
query12 187 115 111 111
query13 1671 557 441 441
query14 11386 9288 9285 9285
query15 195 190 171 171
query16 7669 692 511 511
query17 1228 736 606 606
query18 2042 420 338 338
query19 217 246 174 174
query20 126 123 129 123
query21 214 133 119 119
query22 4110 4012 4056 4012
query23 34142 33113 33157 33113
query24 8428 2390 2412 2390
query25 597 519 462 462
query26 1222 281 162 162
query27 2735 491 368 368
query28 4362 2191 2179 2179
query29 793 619 494 494
query30 301 227 198 198
query31 960 813 724 724
query32 86 78 73 73
query33 598 394 336 336
query34 791 860 528 528
query35 805 823 741 741
query36 956 987 911 911
query37 127 110 85 85
query38 3598 3571 3568 3568
query39 1499 1432 1397 1397
query40 229 139 119 119
query41 66 67 67 67
query42 124 114 113 113
query43 500 492 461 461
query44 1264 786 773 773
query45 191 178 184 178
query46 890 979 652 652
query47 1724 1758 1711 1711
query48 418 429 326 326
query49 817 469 414 414
query50 669 680 406 406
query51 3906 3988 3899 3899
query52 117 112 104 104
query53 251 271 196 196
query54 308 326 280 280
query55 98 87 85 85
query56 336 335 306 306
query57 1152 1171 1123 1123
query58 291 283 287 283
query59 2615 2603 2509 2509
query60 386 369 354 354
query61 193 191 190 190
query62 787 715 670 670
query63 232 200 200 200
query64 4629 1344 1009 1009
query65 4049 3926 3983 3926
query66 1120 461 350 350
query67 15216 15029 15062 15029
query68 7237 974 645 645
query69 510 334 298 298
query70 1366 1252 1238 1238
query71 429 359 333 333
query72 6086 5076 4833 4833
query73 641 578 371 371
query74 8834 8999 8705 8705
query75 3296 3348 2805 2805
query76 3312 1163 752 752
query77 522 447 327 327
query78 9587 9656 8901 8901
query79 2589 837 619 619
query80 768 561 499 499
query81 524 264 225 225
query82 403 162 140 140
query83 275 260 250 250
query84 252 116 97 97
query85 907 494 448 448
query86 407 324 278 278
query87 3763 3732 3637 3637
query88 3825 2273 2268 2268
query89 389 330 296 296
query90 2015 220 223 220
query91 178 172 135 135
query92 89 77 67 67
query93 2586 1014 671 671
query94 751 454 345 345
query95 409 314 318 314
query96 484 584 289 289
query97 2874 2944 2890 2890
query98 238 219 211 211
query99 1283 1391 1316 1316
Total cold run time: 275732 ms
Total hot run time: 188375 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3557688600
# BE UT Coverage Report
Increment line coverage `100.00% (45/45)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/0a24ddd7b4ed3fe7f060f4c675e12b7655f3b084_0a24ddd7b4ed3fe7f060f4c675e12b7655f3b084/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/0a24ddd7b4ed3fe7f060f4c675e12b7655f3b084_0a24ddd7b4ed3fe7f060f4c675e12b7655f3b084/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.63% (18281/34734) |
| Line Coverage | 38.07% (166299/436835) |
| Region Coverage | 33.02% (129306/391620) |
| Branch Coverage | 33.81% (55494/164116) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3555734911
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (45/45)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58002_b5ae6748c1142931e756f7ec6898b28a8308bef4_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58002_b5ae6748c1142931e756f7ec6898b28a8308bef4_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.49% (24349/34058) |
| Line Coverage | 57.95% (252968/436534) |
| Region Coverage | 53.07% (210367/396407) |
| Branch Coverage | 54.56% (89982/164920) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3551593155
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3527896259
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3555406446
<details>
<summary>ClickBench: <b>Total hot run time: 27.92 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b5ae6748c1142931e756f7ec6898b28a8308bef4, data reload: false
query1 0.05 0.05 0.04
query2 0.09 0.05 0.05
query3 0.25 0.09 0.08
query4 1.61 0.12 0.10
query5 0.28 0.26 0.26
query6 1.17 0.65 0.65
query7 0.03 0.03 0.02
query8 0.05 0.05 0.04
query9 0.59 0.53 0.51
query10 0.57 0.58 0.57
query11 0.16 0.11 0.12
query12 0.15 0.12 0.13
query13 0.61 0.60 0.60
query14 1.02 1.01 0.99
query15 0.84 0.83 0.82
query16 0.40 0.40 0.38
query17 0.99 1.02 1.00
query18 0.23 0.20 0.20
query19 1.85 1.80 1.84
query20 0.02 0.01 0.02
query21 15.41 0.20 0.12
query22 5.08 0.07 0.04
query23 15.72 0.27 0.10
query24 2.29 1.28 0.93
query25 0.09 0.06 0.06
query26 0.15 0.13 0.14
query27 0.10 0.06 0.06
query28 4.75 1.21 0.96
query29 12.63 3.91 3.23
query30 0.30 0.14 0.12
query31 2.83 0.58 0.39
query32 3.23 0.54 0.47
query33 3.02 3.01 3.04
query34 15.94 5.17 4.58
query35 4.55 4.58 4.53
query36 0.69 0.50 0.48
query37 0.09 0.08 0.06
query38 0.06 0.04 0.04
query39 0.04 0.04 0.03
query40 0.17 0.14 0.14
query41 0.09 0.04 0.03
query42 0.04 0.03 0.04
query43 0.04 0.04 0.03
Total cold run time: 98.27 s
Total hot run time: 27.92 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3529341621
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (38/38)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58002_fff4f71efb471be7e168f204cde7f85c12485c75_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58002_fff4f71efb471be7e168f204cde7f85c12485c75_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.56% (24346/34021) |
| Line Coverage | 58.04% (253153/436133) |
| Region Coverage | 53.35% (210784/395084) |
| Branch Coverage | 54.68% (90027/164641) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58002:
URL: https://github.com/apache/doris/pull/58002#discussion_r2523640331
##########
be/src/vec/data_types/serde/data_type_number_serde.cpp:
##########
@@ -221,8 +221,15 @@ Status DataTypeNumberSerDe<T>::read_column_from_arrow(IColumn& column,
for (size_t offset_i = start; offset_i < end; ++offset_i) {
if (!concrete_array->IsNull(offset_i)) {
- const auto* raw_data = buffer->data() + concrete_array->value_offset(offset_i);
- const auto raw_data_len = concrete_array->value_length(offset_i);
+ int32_t start_offset;
Review Comment:
变量一定得初始化
##########
be/src/vec/data_types/serde/data_type_datetimev2_serde.cpp:
##########
@@ -398,7 +398,11 @@ Status DataTypeDateTimeV2SerDe::read_column_from_arrow(IColumn& column,
}
}
for (auto value_i = start; value_i < end; ++value_i) {
- auto utc_epoch = static_cast<UInt64>(concrete_array->Value(value_i));
+ int64_t utc_epoch_raw;
Review Comment:
这些有单测吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "Chen768959 (via GitHub)" <gi...@apache.org>.
Chen768959 commented on code in PR #58002:
URL: https://github.com/apache/doris/pull/58002#discussion_r2523902180
##########
be/src/vec/data_types/serde/data_type_datetimev2_serde.cpp:
##########
@@ -398,7 +398,11 @@ Status DataTypeDateTimeV2SerDe::read_column_from_arrow(IColumn& column,
}
}
for (auto value_i = start; value_i < end; ++value_i) {
- auto utc_epoch = static_cast<UInt64>(concrete_array->Value(value_i));
+ int64_t utc_epoch_raw;
Review Comment:
我补充一下，我想办法构造下各类型的内存未对齐的arrow数据场景
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3555626324
# BE UT Coverage Report
Increment line coverage `100.00% (45/45)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/b5ae6748c1142931e756f7ec6898b28a8308bef4_b5ae6748c1142931e756f7ec6898b28a8308bef4/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/b5ae6748c1142931e756f7ec6898b28a8308bef4_b5ae6748c1142931e756f7ec6898b28a8308bef4/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.74% (18279/34658) |
| Line Coverage | 38.12% (166172/435865) |
| Region Coverage | 33.08% (129276/390831) |
| Branch Coverage | 33.84% (55437/163841) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3556665768
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3551593283
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3555355678
<details>
<summary>TPC-H: <b>Total hot run time: 34297 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b5ae6748c1142931e756f7ec6898b28a8308bef4, data reload: false
------ Round 1 ----------------------------------
q1 17649 5044 4881 4881
q2 2055 310 213 213
q3 10271 1279 683 683
q4 10231 937 380 380
q5 7517 2385 2404 2385
q6 193 171 138 138
q7 914 776 633 633
q8 9346 1423 1168 1168
q9 7248 5412 5406 5406
q10 6899 2248 1794 1794
q11 515 301 299 299
q12 366 361 226 226
q13 17797 3635 3005 3005
q14 231 229 219 219
q15 591 515 505 505
q16 1035 1015 940 940
q17 597 879 356 356
q18 7583 7218 6948 6948
q19 1104 948 550 550
q20 359 377 231 231
q21 3722 2494 2341 2341
q22 1084 1050 996 996
Total cold run time: 107307 ms
Total hot run time: 34297 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4908 4968 4930 4930
q2 331 415 315 315
q3 2214 2669 2268 2268
q4 1318 1780 1344 1344
q5 4182 4390 4455 4390
q6 218 181 133 133
q7 2016 1987 1816 1816
q8 2634 2610 2484 2484
q9 7654 7567 7714 7567
q10 3008 3315 2742 2742
q11 619 531 497 497
q12 670 784 642 642
q13 3505 3909 3352 3352
q14 285 313 294 294
q15 554 538 534 534
q16 1052 1117 1065 1065
q17 1242 1629 1364 1364
q18 7760 7639 7882 7639
q19 802 841 1012 841
q20 2006 2062 1955 1955
q21 4765 4318 4293 4293
q22 1083 1066 990 990
Total cold run time: 52826 ms
Total hot run time: 51455 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3555182288
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3528559660
<details>
<summary>ClickBench: <b>Total hot run time: 27.38 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fff4f71efb471be7e168f204cde7f85c12485c75, data reload: false
query1 0.06 0.05 0.05
query2 0.09 0.05 0.05
query3 0.26 0.09 0.09
query4 1.61 0.12 0.11
query5 0.28 0.25 0.25
query6 1.18 0.65 0.64
query7 0.03 0.02 0.03
query8 0.06 0.04 0.04
query9 0.58 0.52 0.52
query10 0.58 0.57 0.57
query11 0.17 0.11 0.12
query12 0.15 0.12 0.12
query13 0.63 0.61 0.60
query14 1.01 1.01 1.00
query15 0.86 0.84 0.82
query16 0.39 0.40 0.42
query17 1.02 1.06 1.05
query18 0.22 0.20 0.20
query19 1.88 1.83 1.76
query20 0.01 0.02 0.01
query21 15.44 0.20 0.13
query22 5.10 0.07 0.05
query23 15.67 0.27 0.10
query24 2.45 0.95 0.29
query25 0.07 0.07 0.06
query26 0.14 0.14 0.13
query27 0.07 0.06 0.05
query28 4.12 1.13 0.95
query29 12.58 4.00 3.29
query30 0.29 0.14 0.12
query31 2.82 0.59 0.37
query32 3.22 0.56 0.48
query33 3.03 3.03 3.05
query34 15.90 5.13 4.50
query35 4.64 4.61 4.57
query36 0.67 0.50 0.49
query37 0.09 0.07 0.07
query38 0.06 0.05 0.04
query39 0.03 0.04 0.03
query40 0.18 0.14 0.13
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 97.81 s
Total hot run time: 27.38 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "Chen768959 (via GitHub)" <gi...@apache.org>.
Chen768959 commented on code in PR #58002:
URL: https://github.com/apache/doris/pull/58002#discussion_r2523878160
##########
be/src/vec/data_types/serde/data_type_number_serde.cpp:
##########
@@ -221,8 +221,15 @@ Status DataTypeNumberSerDe<T>::read_column_from_arrow(IColumn& column,
for (size_t offset_i = start; offset_i < end; ++offset_i) {
if (!concrete_array->IsNull(offset_i)) {
- const auto* raw_data = buffer->data() + concrete_array->value_offset(offset_i);
- const auto raw_data_len = concrete_array->value_length(offset_i);
+ int32_t start_offset;
Review Comment:
好的，我都添加一下
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3528288146
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3555132382
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3555397674
<details>
<summary>TPC-DS: <b>Total hot run time: 187936 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b5ae6748c1142931e756f7ec6898b28a8308bef4, data reload: false
query1 1024 415 427 415
query2 6577 1732 1670 1670
query3 6758 231 231 231
query4 25891 23333 22675 22675
query5 4359 645 537 537
query6 326 251 229 229
query7 4655 498 304 304
query8 301 263 266 263
query9 8687 2959 2968 2959
query10 489 397 301 301
query11 15577 15117 14728 14728
query12 176 120 116 116
query13 1678 589 468 468
query14 10191 9153 9199 9153
query15 198 187 171 171
query16 7150 661 477 477
query17 982 782 658 658
query18 2013 435 349 349
query19 220 217 186 186
query20 133 134 130 130
query21 215 136 117 117
query22 4002 4112 3924 3924
query23 33842 33076 33053 33053
query24 8151 2412 2463 2412
query25 646 549 505 505
query26 1264 278 168 168
query27 2741 510 379 379
query28 4434 2282 2256 2256
query29 871 650 537 537
query30 295 226 204 204
query31 897 802 723 723
query32 102 87 84 84
query33 607 415 364 364
query34 775 861 539 539
query35 823 840 766 766
query36 957 977 917 917
query37 141 127 96 96
query38 3582 3519 3463 3463
query39 1488 1435 1419 1419
query40 234 150 129 129
query41 68 65 64 64
query42 135 120 121 120
query43 503 481 463 463
query44 1268 828 823 823
query45 198 186 177 177
query46 889 996 667 667
query47 1740 1813 1712 1712
query48 401 458 346 346
query49 788 530 437 437
query50 658 683 427 427
query51 3894 3904 3841 3841
query52 118 119 121 119
query53 265 283 209 209
query54 351 333 326 326
query55 100 97 96 96
query56 369 411 353 353
query57 1191 1173 1123 1123
query58 302 288 287 287
query59 2548 2775 2438 2438
query60 367 372 352 352
query61 154 170 164 164
query62 797 707 635 635
query63 231 196 199 196
query64 4602 1207 897 897
query65 4043 3906 3921 3906
query66 1186 456 349 349
query67 15064 15088 14717 14717
query68 8345 947 640 640
query69 498 342 312 312
query70 1333 1294 1206 1206
query71 480 416 342 342
query72 5923 4883 4954 4883
query73 689 635 376 376
query74 8969 8790 8758 8758
query75 3872 3259 2805 2805
query76 3694 1112 756 756
query77 828 413 336 336
query78 9511 9507 8939 8939
query79 2790 834 615 615
query80 715 593 533 533
query81 504 256 227 227
query82 478 166 143 143
query83 271 270 255 255
query84 303 116 98 98
query85 940 515 461 461
query86 399 318 295 295
query87 3659 3733 3631 3631
query88 3737 2258 2231 2231
query89 394 328 310 310
query90 1902 246 237 237
query91 176 167 139 139
query92 96 82 80 80
query93 2214 1033 701 701
query94 717 427 329 329
query95 427 348 344 344
query96 490 568 284 284
query97 2913 2971 2856 2856
query98 258 227 212 212
query99 1398 1414 1283 1283
Total cold run time: 274605 ms
Total hot run time: 187936 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #58002:
URL: https://github.com/apache/doris/pull/58002
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3557098601
<details>
<summary>TPC-H: <b>Total hot run time: 34334 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0a24ddd7b4ed3fe7f060f4c675e12b7655f3b084, data reload: false
------ Round 1 ----------------------------------
q1 17634 5061 4916 4916
q2 2072 313 205 205
q3 10240 1296 723 723
q4 10230 961 389 389
q5 7505 2449 2286 2286
q6 191 174 146 146
q7 917 765 643 643
q8 9340 1373 1115 1115
q9 7231 5345 5377 5345
q10 6909 2258 1825 1825
q11 493 297 284 284
q12 373 365 229 229
q13 17784 3709 3015 3015
q14 236 237 216 216
q15 594 519 500 500
q16 1016 989 946 946
q17 596 875 360 360
q18 7665 7197 7122 7122
q19 1153 953 549 549
q20 354 343 233 233
q21 3697 3241 2327 2327
q22 1054 1017 960 960
Total cold run time: 107284 ms
Total hot run time: 34334 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5009 4947 4947 4947
q2 333 419 322 322
q3 2198 2687 2369 2369
q4 1351 1764 1358 1358
q5 4205 4503 4608 4503
q6 219 185 142 142
q7 2160 1939 1851 1851
q8 2636 2693 2586 2586
q9 7811 7431 7660 7431
q10 3111 3389 2832 2832
q11 591 550 520 520
q12 680 778 668 668
q13 3574 3989 3336 3336
q14 289 313 277 277
q15 554 515 504 504
q16 1107 1146 1103 1103
q17 1181 1623 1440 1440
q18 7864 7678 7566 7566
q19 825 818 1083 818
q20 2047 2126 1898 1898
q21 5245 4613 4301 4301
q22 1108 1028 999 999
Total cold run time: 54098 ms
Total hot run time: 51771 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3557197407
<details>
<summary>ClickBench: <b>Total hot run time: 27.89 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0a24ddd7b4ed3fe7f060f4c675e12b7655f3b084, data reload: false
query1 0.06 0.05 0.05
query2 0.10 0.05 0.04
query3 0.26 0.09 0.09
query4 1.61 0.12 0.11
query5 0.27 0.25 0.24
query6 1.16 0.65 0.65
query7 0.03 0.03 0.03
query8 0.05 0.04 0.04
query9 0.58 0.53 0.53
query10 0.58 0.58 0.57
query11 0.16 0.12 0.11
query12 0.15 0.12 0.12
query13 0.63 0.60 0.61
query14 1.01 0.99 1.00
query15 0.85 0.85 0.83
query16 0.41 0.40 0.40
query17 1.03 1.00 0.99
query18 0.22 0.20 0.20
query19 1.88 1.75 1.83
query20 0.02 0.01 0.01
query21 15.44 0.17 0.14
query22 5.15 0.08 0.05
query23 15.64 0.25 0.10
query24 3.08 1.12 0.64
query25 0.08 0.06 0.06
query26 0.15 0.13 0.13
query27 0.06 0.06 0.06
query28 4.95 1.16 0.94
query29 12.62 3.89 3.33
query30 0.30 0.14 0.12
query31 2.82 0.61 0.39
query32 3.24 0.55 0.47
query33 3.00 3.07 3.11
query34 15.86 5.25 4.57
query35 4.56 4.62 4.61
query36 0.68 0.50 0.51
query37 0.10 0.07 0.07
query38 0.07 0.04 0.04
query39 0.04 0.03 0.03
query40 0.18 0.14 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.02
query43 0.05 0.03 0.04
Total cold run time: 99.25 s
Total hot run time: 27.89 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3557937509
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (45/45)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58002_0a24ddd7b4ed3fe7f060f4c675e12b7655f3b084_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58002_0a24ddd7b4ed3fe7f060f4c675e12b7655f3b084_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.51% (24375/34088) |
| Line Coverage | 57.96% (253292/436975) |
| Region Coverage | 53.17% (211029/396921) |
| Branch Coverage | 54.55% (90071/165127) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3528480012
<details>
<summary>TPC-H: <b>Total hot run time: 33977 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fff4f71efb471be7e168f204cde7f85c12485c75, data reload: false
------ Round 1 ----------------------------------
q1 18749 5442 5010 5010
q2 2023 313 200 200
q3 10272 1425 708 708
q4 10224 874 367 367
q5 7507 2401 2367 2367
q6 183 167 137 137
q7 913 758 650 650
q8 9379 1281 1085 1085
q9 6999 5248 5114 5114
q10 6853 2243 1783 1783
q11 487 295 287 287
q12 336 364 230 230
q13 17791 3641 3022 3022
q14 233 237 205 205
q15 578 529 496 496
q16 972 998 938 938
q17 598 865 361 361
q18 7317 7128 7007 7007
q19 1200 952 542 542
q20 362 336 224 224
q21 3689 3208 2288 2288
q22 1024 1047 956 956
Total cold run time: 107689 ms
Total hot run time: 33977 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5156 5085 5064 5064
q2 250 327 225 225
q3 2143 2631 2347 2347
q4 1357 1774 1338 1338
q5 4176 4442 4525 4442
q6 208 171 130 130
q7 1920 2013 1870 1870
q8 2668 2491 2659 2491
q9 7355 7412 7361 7361
q10 3064 3249 2765 2765
q11 573 542 499 499
q12 708 787 576 576
q13 3588 3900 3403 3403
q14 304 299 282 282
q15 680 554 517 517
q16 1058 1067 1076 1067
q17 1160 1483 1393 1393
q18 8035 7650 7810 7650
q19 798 793 894 793
q20 1998 2113 1921 1921
q21 4808 4413 4404 4404
q22 1072 1067 1003 1003
Total cold run time: 53079 ms
Total hot run time: 51541 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3528537487
<details>
<summary>TPC-DS: <b>Total hot run time: 187925 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fff4f71efb471be7e168f204cde7f85c12485c75, data reload: false
query1 1032 402 410 402
query2 6570 1681 1723 1681
query3 6757 223 224 223
query4 26085 23839 23117 23117
query5 4422 637 505 505
query6 338 261 234 234
query7 4654 503 295 295
query8 338 257 260 257
query9 8710 2603 2601 2601
query10 493 336 293 293
query11 15873 15094 14796 14796
query12 176 129 117 117
query13 1667 568 442 442
query14 11090 9139 9174 9139
query15 197 188 181 181
query16 7371 655 462 462
query17 1230 764 649 649
query18 2000 431 335 335
query19 213 210 183 183
query20 138 130 125 125
query21 221 134 117 117
query22 3967 4134 4050 4050
query23 34075 33019 33164 33019
query24 8427 2452 2437 2437
query25 655 542 481 481
query26 1250 277 169 169
query27 2744 498 360 360
query28 4368 2240 2201 2201
query29 867 636 501 501
query30 308 229 200 200
query31 883 796 739 739
query32 85 77 77 77
query33 612 375 346 346
query34 803 854 534 534
query35 821 835 758 758
query36 960 992 912 912
query37 128 119 89 89
query38 3593 3517 3433 3433
query39 1470 1422 1434 1422
query40 229 134 125 125
query41 68 65 64 64
query42 126 117 114 114
query43 482 499 474 474
query44 1244 804 780 780
query45 189 182 176 176
query46 893 1001 655 655
query47 1757 1817 1698 1698
query48 407 430 337 337
query49 783 539 447 447
query50 637 691 415 415
query51 3862 3982 3942 3942
query52 114 115 101 101
query53 243 268 198 198
query54 309 299 289 289
query55 95 89 82 82
query56 323 323 318 318
query57 1175 1190 1117 1117
query58 278 276 269 269
query59 2557 2745 2600 2600
query60 335 357 327 327
query61 162 153 157 153
query62 787 725 661 661
query63 233 199 194 194
query64 4475 1152 917 917
query65 4002 3913 3942 3913
query66 1160 435 344 344
query67 15139 15188 15024 15024
query68 8556 885 600 600
query69 517 319 296 296
query70 1374 1253 1294 1253
query71 510 346 313 313
query72 5919 4917 4971 4917
query73 717 606 360 360
query74 9158 8728 8925 8728
query75 3916 3339 2836 2836
query76 3778 1158 729 729
query77 832 405 320 320
query78 9576 9920 8857 8857
query79 1931 838 599 599
query80 634 583 527 527
query81 496 268 236 236
query82 483 162 129 129
query83 272 265 250 250
query84 254 117 88 88
query85 987 473 432 432
query86 336 323 302 302
query87 3728 3740 3586 3586
query88 3305 2269 2231 2231
query89 386 337 298 298
query90 1968 226 216 216
query91 168 174 138 138
query92 80 71 67 67
query93 1159 973 654 654
query94 702 461 355 355
query95 413 323 316 316
query96 485 577 282 282
query97 2904 2986 2872 2872
query98 240 213 213 213
query99 1329 1394 1309 1309
Total cold run time: 274768 ms
Total hot run time: 187925 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](datatype)Fix unaligned memory access in read_column_from_arrow [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58002:
URL: https://github.com/apache/doris/pull/58002#issuecomment-3528605583
# BE UT Coverage Report
Increment line coverage `100.00% (38/38)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/fff4f71efb471be7e168f204cde7f85c12485c75_fff4f71efb471be7e168f204cde7f85c12485c75/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/fff4f71efb471be7e168f204cde7f85c12485c75_fff4f71efb471be7e168f204cde7f85c12485c75/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.81% (18283/34621) |
| Line Coverage | 38.18% (166253/435457) |
| Region Coverage | 33.16% (129170/389573) |
| Branch Coverage | 33.90% (55451/163566) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org