You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "liutang123 (via GitHub)" <gi...@apache.org> on 2025/05/23 08:33:09 UTC
[PR] [Fix](multicatelog) Fix insert iceberg/hive table when use broker [doris]
liutang123 opened a new pull request, #51187:
URL: https://github.com/apache/doris/pull/51187
### What problem does this PR solve?
Issue Number: close #xxx
```
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) in /usr/local/service/doris/lib/be/doris_be
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.10] in /usr/local/jdk17/lib/server/libjvm.so
2# JVM_handle_linux_signal in /usr/local/jdk17/lib/server/libjvm.so
3# 0x00007F1AD2254AD0 in /lib64/libc.so.6
4# doris::FileFactory::create_fs(doris::io::FSPropertiesRef const&, doris::io::FileDescription const&) in /usr/local/service/doris/lib/be/doris_be
5# doris::vectorized::VIcebergPartitionWriter::open(doris::RuntimeState*, doris::RuntimeProfile*) in /usr/local/service/doris/lib/be/doris_be
6# doris::vectorized::VIcebergTableWriter::write(doris::RuntimeState*, doris::vectorized::Block&) in /usr/local/service/doris/lib/be/doris_be
7# doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) in /usr/local/service/doris/lib/be/doris_be
8# std::_Function_handler<void (), doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0>::_M_invoke(std::_Any_data const&) in /usr/local/service/doris/lib/be/doris_be
```
Related PR: #xxx
Problem Summary:
When write Iceberg/Hive table and use broker, FE should pass broker list to BE.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](multicatelog) Fix insert iceberg/hive table when use broker [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #51187:
URL: https://github.com/apache/doris/pull/51187#issuecomment-2903701326
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](multicatelog) Fix insert iceberg/hive table when use broker [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51187:
URL: https://github.com/apache/doris/pull/51187#issuecomment-2905133411
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6674f9922dfd07c53e4f3189d5ae80de0a3a5e13_6674f9922dfd07c53e4f3189d5ae80de0a3a5e13_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6674f9922dfd07c53e4f3189d5ae80de0a3a5e13_6674f9922dfd07c53e4f3189d5ae80de0a3a5e13_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 83.25% (1113/1337) |
| Line Coverage | 66.18% (18680/28224) |
| Region Coverage | 65.85% (9271/14079) |
| Branch Coverage | 55.60% (4985/8966) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](multicatelog) Fix insert iceberg/hive table when use broker [doris]
Posted by "liutang123 (via GitHub)" <gi...@apache.org>.
liutang123 merged PR #51187:
URL: https://github.com/apache/doris/pull/51187
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](multicatelog) Fix insert iceberg/hive table when use broker [doris]
Posted by "liutang123 (via GitHub)" <gi...@apache.org>.
liutang123 commented on PR #51187:
URL: https://github.com/apache/doris/pull/51187#issuecomment-2905052182
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](multicatelog) Fix insert iceberg/hive table when use broker [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51187:
URL: https://github.com/apache/doris/pull/51187#issuecomment-2905520883
<details>
<summary>TPC-DS: <b>Total hot run time: 192268 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6674f9922dfd07c53e4f3189d5ae80de0a3a5e13, data reload: false
query1 1430 1069 1062 1062
query2 6295 1823 1832 1823
query3 11005 4475 4390 4390
query4 55297 25226 23235 23235
query5 5149 491 467 467
query6 374 212 186 186
query7 4904 508 292 292
query8 313 242 230 230
query9 5904 2624 2625 2624
query10 450 323 273 273
query11 15092 14991 14814 14814
query12 168 115 106 106
query13 1052 540 417 417
query14 10204 6251 6259 6251
query15 202 197 185 185
query16 7131 655 526 526
query17 1059 730 590 590
query18 1514 433 314 314
query19 200 222 164 164
query20 118 126 127 126
query21 214 124 107 107
query22 4492 4411 4319 4319
query23 34026 33326 33616 33326
query24 6674 2418 2423 2418
query25 484 504 442 442
query26 699 278 165 165
query27 2199 517 358 358
query28 3009 2178 2144 2144
query29 600 596 476 476
query30 274 227 194 194
query31 858 875 777 777
query32 76 66 67 66
query33 460 379 336 336
query34 785 874 530 530
query35 787 885 737 737
query36 938 1045 919 919
query37 122 101 85 85
query38 4213 4292 4155 4155
query39 1561 1496 1506 1496
query40 212 120 109 109
query41 54 55 53 53
query42 133 117 113 113
query43 512 536 491 491
query44 1343 851 841 841
query45 184 174 166 166
query46 856 1028 656 656
query47 1884 1885 1804 1804
query48 407 455 344 344
query49 697 515 431 431
query50 678 705 409 409
query51 4216 4251 4271 4251
query52 114 111 103 103
query53 227 262 199 199
query54 614 586 529 529
query55 92 90 87 87
query56 326 319 301 301
query57 1151 1202 1163 1163
query58 271 265 275 265
query59 2731 2821 2701 2701
query60 337 331 326 326
query61 129 126 125 125
query62 705 760 679 679
query63 227 189 198 189
query64 1485 1048 666 666
query65 4338 4216 4184 4184
query66 728 407 303 303
query67 16212 15692 15516 15516
query68 7237 897 541 541
query69 560 311 287 287
query70 1196 1130 1062 1062
query71 489 329 306 306
query72 5527 4727 4854 4727
query73 1381 647 361 361
query74 8928 9180 8652 8652
query75 3698 3212 2690 2690
query76 4264 1205 744 744
query77 632 371 294 294
query78 10019 10068 9230 9230
query79 2532 804 588 588
query80 651 513 443 443
query81 485 253 221 221
query82 470 126 93 93
query83 377 252 237 237
query84 292 121 85 85
query85 797 394 312 312
query86 404 286 283 283
query87 4500 4413 4384 4384
query88 3612 2313 2304 2304
query89 410 317 279 279
query90 1907 210 216 210
query91 140 137 110 110
query92 70 58 58 58
query93 1409 959 593 593
query94 668 409 311 311
query95 372 288 289 288
query96 497 557 277 277
query97 2709 2772 2669 2669
query98 231 221 211 211
query99 1453 1396 1289 1289
Total cold run time: 299112 ms
Total hot run time: 192268 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](multicatelog) Fix insert iceberg/hive table when use broker [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51187:
URL: https://github.com/apache/doris/pull/51187#issuecomment-2906805796
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](multicatelog) Fix insert iceberg/hive table when use broker [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51187:
URL: https://github.com/apache/doris/pull/51187#issuecomment-2906805814
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](multicatelog) Fix insert iceberg/hive table when use broker [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51187:
URL: https://github.com/apache/doris/pull/51187#issuecomment-2905492621
<details>
<summary>TPC-H: <b>Total hot run time: 33728 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6674f9922dfd07c53e4f3189d5ae80de0a3a5e13, data reload: false
------ Round 1 ----------------------------------
q1 26417 5006 4955 4955
q2 2083 284 178 178
q3 10502 1228 704 704
q4 10236 993 517 517
q5 7655 2370 2321 2321
q6 185 162 131 131
q7 922 710 609 609
q8 9319 1492 1078 1078
q9 6846 5155 5105 5105
q10 6868 2318 1918 1918
q11 504 287 288 287
q12 348 352 210 210
q13 17795 3629 3083 3083
q14 225 243 219 219
q15 545 495 488 488
q16 436 434 377 377
q17 591 856 376 376
q18 7711 7193 7095 7095
q19 2064 1039 559 559
q20 331 329 222 222
q21 3677 3161 2355 2355
q22 1066 1021 941 941
Total cold run time: 116326 ms
Total hot run time: 33728 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5200 5060 5039 5039
q2 230 323 239 239
q3 2168 2676 2296 2296
q4 1355 1774 1412 1412
q5 4468 4393 4408 4393
q6 219 178 131 131
q7 2001 1939 1786 1786
q8 2570 2491 2605 2491
q9 7275 7238 7019 7019
q10 3042 3185 2754 2754
q11 568 512 482 482
q12 683 745 609 609
q13 3526 3876 3312 3312
q14 278 295 275 275
q15 527 476 472 472
q16 459 466 443 443
q17 1140 1567 1371 1371
q18 7947 7567 7488 7488
q19 790 874 1086 874
q20 2001 2029 1853 1853
q21 4711 4415 4552 4415
q22 1107 1068 1038 1038
Total cold run time: 52265 ms
Total hot run time: 50192 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](multicatelog) Fix insert iceberg/hive table when use broker [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51187:
URL: https://github.com/apache/doris/pull/51187#issuecomment-2905540623
<details>
<summary>ClickBench: <b>Total hot run time: 29.62 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6674f9922dfd07c53e4f3189d5ae80de0a3a5e13, data reload: false
query1 0.04 0.04 0.03
query2 0.15 0.11 0.10
query3 0.34 0.20 0.20
query4 1.59 0.21 0.20
query5 0.48 0.45 0.46
query6 1.21 0.65 0.66
query7 0.02 0.02 0.02
query8 0.06 0.05 0.05
query9 0.62 0.53 0.52
query10 0.58 0.60 0.57
query11 0.25 0.15 0.12
query12 0.24 0.14 0.13
query13 0.63 0.61 0.62
query14 0.78 0.82 0.82
query15 0.96 0.87 0.87
query16 0.36 0.38 0.37
query17 1.05 1.05 1.05
query18 0.17 0.18 0.20
query19 1.98 1.89 1.84
query20 0.02 0.01 0.02
query21 15.40 0.96 0.66
query22 0.93 1.07 0.79
query23 14.72 1.54 0.74
query24 5.44 0.57 0.29
query25 0.17 0.10 0.08
query26 0.55 0.21 0.18
query27 0.09 0.09 0.09
query28 11.04 1.19 0.58
query29 12.54 4.10 3.39
query30 0.28 0.08 0.07
query31 2.84 0.66 0.43
query32 3.23 0.60 0.51
query33 3.05 3.05 3.16
query34 16.67 5.13 4.40
query35 4.49 4.43 4.42
query36 0.63 0.51 0.49
query37 0.20 0.18 0.17
query38 0.16 0.15 0.16
query39 0.05 0.04 0.04
query40 0.18 0.17 0.16
query41 0.10 0.06 0.05
query42 0.06 0.05 0.05
query43 0.06 0.05 0.04
Total cold run time: 104.41 s
Total hot run time: 29.62 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org