You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2024/06/18 04:05:18 UTC
[PR] [chore](routine-load) optimize out of range error message [doris]
HHoflittlefish777 opened a new pull request, #36450:
URL: https://github.com/apache/doris/pull/36450
## Proposed changes
before
```
ErrorReason{code=errCode = 105, msg='be 10002 abort task, task id: d846f3d3-7c9e-44a7-bee0-3eff8cd11c6f job id: 11310 with reason: [INTERNAL_ERROR]Offset out of range,
0# doris::Status doris::Status::Error<6, true>(std::basic_string_view<char, std::char_traits<char> >) at /mnt/disk1/laihui/doris/be/src/common/status.h:422
1# doris::Status doris::Status::InternalError<true>(std::basic_string_view<char, std::char_traits<char> >) at /mnt/disk1/laihui/doris/be/src/common/status.h:468
2# doris::KafkaDataConsumer::group_consume(doris::BlockingQueue<RdKafka::Message*>*, long) at /mnt/disk1/laihui/doris/be/src/runtime/routine_load/data_consumer.cpp:226
3# doris::KafkaDataConsumerGroup::actual_consume(std::shared_ptr<doris::DataConsumer>, doris::BlockingQueue<RdKafka::Message*>*, long, std::function<void (doris::Status const&)>) at /mnt/disk1/laihui/doris/be/src/runtime/routine_load/data_consumer_group.cpp:200
4# void std::__invoke_impl<void, void (doris::KafkaDataConsumerGroup::*&)(std::shared_ptr<doris::DataConsumer>, doris::BlockingQueue<RdKafka::Message*>*, long, std::function<void (doris::Status const&)>), doris::KafkaDataConsumerGroup*&, std::shared_ptr<doris::DataConsumer>&, doris::BlockingQueue<RdKafka::Message*>*&, long&, doris::KafkaDataConsumerGroup::start_all(std::shared_ptr<doris::StreamLoadContext>, std::shared_ptr<doris::io::KafkaConsumerPipe>)::$_0&>(std::__invoke_memfun_deref, void (doris::KafkaDataConsumerGroup::*&)(std::shared_ptr<doris::DataConsumer>, doris::BlockingQueue<RdKafka::Message*>*, long, std::function<void (doris::Status const&)>), doris::KafkaDataConsumerGroup*&, std::shared_ptr<doris::DataConsumer>&, doris::BlockingQueue<RdKafka::Message*>*&, long&, doris::KafkaDataConsumerGroup::start_all(std::shared_ptr<doris::StreamLoadContext>, std::shared_ptr<doris::io::KafkaConsumerPipe>)::$_0&) at /mnt/disk1/laihui/build/ldb_toolchain/bin/../lib/gcc/x86_6
4-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74
...
```
now
```
ErrorReason{code=errCode = 105, msg='be 10002 abort task, task id: 3ba0c0f4-d13c-4dfa-90ce-3df922fd9340 job id: 11310 with reason: [INTERNAL_ERROR]Offset out of range, consume partition 0, consume offset 100, the offset used by job does not exist in kafka, please check the offset, using the Alter ROUTINE LOAD command to modify it, and resume the job'}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2174949673
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2175422841
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2177417058
<details>
<summary>TPC-DS: <b>Total hot run time: 173979 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 58c2a6cf42d65e56eabed95726f3fee497c8459a, data reload: false
query1 805 374 365 365
query2 6321 2444 2542 2444
query3 6402 205 207 205
query4 18701 17383 17366 17366
query5 3802 474 485 474
query6 247 162 180 162
query7 4466 306 285 285
query8 327 299 294 294
query9 8382 2416 2396 2396
query10 577 322 288 288
query11 10457 10040 9993 9993
query12 135 87 84 84
query13 1624 358 361 358
query14 10252 7707 7553 7553
query15 241 194 191 191
query16 7836 270 257 257
query17 1841 541 535 535
query18 1973 265 272 265
query19 199 147 156 147
query20 89 88 83 83
query21 211 127 130 127
query22 4334 4115 3977 3977
query23 33567 33703 33467 33467
query24 10565 2783 2803 2783
query25 585 352 348 348
query26 700 151 149 149
query27 2217 317 318 317
query28 5554 2061 2058 2058
query29 854 640 627 627
query30 249 152 157 152
query31 945 729 754 729
query32 97 54 53 53
query33 661 270 294 270
query34 857 478 471 471
query35 716 633 592 592
query36 1100 930 939 930
query37 132 69 71 69
query38 2868 2720 2721 2720
query39 875 795 788 788
query40 207 128 122 122
query41 58 51 53 51
query42 126 98 100 98
query43 595 568 567 567
query44 1061 738 732 732
query45 192 162 157 157
query46 1070 717 712 712
query47 1888 1775 1790 1775
query48 373 288 288 288
query49 870 409 408 408
query50 755 392 380 380
query51 6715 6646 6667 6646
query52 107 94 94 94
query53 355 289 287 287
query54 873 458 438 438
query55 74 75 77 75
query56 291 272 278 272
query57 1176 1042 1031 1031
query58 270 252 255 252
query59 3661 3209 3045 3045
query60 305 296 292 292
query61 162 93 94 93
query62 605 432 462 432
query63 322 290 280 280
query64 8556 2207 1769 1769
query65 3161 3114 3127 3114
query66 790 321 323 321
query67 15621 15089 15093 15089
query68 4496 524 527 524
query69 545 428 361 361
query70 1194 1089 1184 1089
query71 387 277 273 273
query72 7121 5793 5452 5452
query73 757 320 314 314
query74 5894 5511 5568 5511
query75 3421 2657 2681 2657
query76 2330 944 920 920
query77 482 296 288 288
query78 10286 9886 9846 9846
query79 2473 512 503 503
query80 2057 456 459 456
query81 602 229 218 218
query82 905 99 100 99
query83 307 168 175 168
query84 321 82 91 82
query85 1259 279 264 264
query86 440 301 311 301
query87 3308 3096 3122 3096
query88 4224 2315 2343 2315
query89 467 385 391 385
query90 1866 186 180 180
query91 123 104 98 98
query92 57 51 49 49
query93 1922 499 507 499
query94 1238 186 185 185
query95 398 315 310 310
query96 583 264 256 256
query97 3236 3065 3066 3065
query98 224 209 189 189
query99 1167 834 859 834
Total cold run time: 266093 ms
Total hot run time: 173979 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2177359429
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2175453310
TeamCity be ut coverage result:
Function Coverage: 36.48% (9003/24679)
Line Coverage: 28.02% (73756/263201)
Region Coverage: 27.49% (38304/139336)
Branch Coverage: 24.19% (19523/80706)
Coverage Report: http://coverage.selectdb-in.cc/coverage/58c2a6cf42d65e56eabed95726f3fee497c8459a_58c2a6cf42d65e56eabed95726f3fee497c8459a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2174968166
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2177422575
<details>
<summary>ClickBench: <b>Total hot run time: 31.01 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 58c2a6cf42d65e56eabed95726f3fee497c8459a, data reload: false
query1 0.04 0.04 0.04
query2 0.07 0.04 0.04
query3 0.23 0.05 0.05
query4 1.67 0.08 0.10
query5 0.50 0.46 0.48
query6 1.13 0.72 0.73
query7 0.02 0.02 0.01
query8 0.05 0.03 0.04
query9 0.54 0.50 0.48
query10 0.55 0.55 0.53
query11 0.15 0.11 0.12
query12 0.14 0.12 0.12
query13 0.59 0.59 0.60
query14 0.77 0.78 0.77
query15 0.83 0.82 0.81
query16 0.37 0.37 0.37
query17 1.03 1.03 1.06
query18 0.22 0.26 0.24
query19 1.90 1.71 1.81
query20 0.02 0.01 0.01
query21 15.40 0.65 0.66
query22 3.80 6.97 2.46
query23 18.28 1.34 1.23
query24 2.08 0.21 0.22
query25 0.17 0.08 0.08
query26 0.25 0.18 0.18
query27 0.07 0.08 0.08
query28 13.24 1.01 0.99
query29 12.69 3.30 3.27
query30 0.26 0.06 0.06
query31 2.87 0.41 0.38
query32 3.27 0.48 0.46
query33 2.91 2.87 2.91
query34 16.88 4.40 4.42
query35 4.46 4.46 4.45
query36 0.65 0.46 0.50
query37 0.18 0.15 0.14
query38 0.16 0.14 0.14
query39 0.05 0.03 0.04
query40 0.17 0.15 0.14
query41 0.10 0.04 0.05
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 108.86 s
Total hot run time: 31.01 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2174968583
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2177406031
<details>
<summary>TPC-H: <b>Total hot run time: 40105 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 58c2a6cf42d65e56eabed95726f3fee497c8459a, data reload: false
------ Round 1 ----------------------------------
q1 17604 4322 4318 4318
q2 2026 188 199 188
q3 10452 1120 1094 1094
q4 10207 798 780 780
q5 7467 2722 2596 2596
q6 218 134 135 134
q7 972 588 587 587
q8 9221 2036 2072 2036
q9 9074 6476 6460 6460
q10 9044 3707 3722 3707
q11 441 240 238 238
q12 498 228 229 228
q13 17764 2977 2987 2977
q14 271 219 209 209
q15 512 468 479 468
q16 528 391 382 382
q17 954 630 734 630
q18 8180 7945 7807 7807
q19 7882 1443 1274 1274
q20 710 326 328 326
q21 5131 3319 4118 3319
q22 419 357 347 347
Total cold run time: 119575 ms
Total hot run time: 40105 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4579 4384 4472 4384
q2 369 268 257 257
q3 3055 2876 2868 2868
q4 1998 1708 1669 1669
q5 5451 5540 5493 5493
q6 225 128 130 128
q7 2192 1812 1905 1812
q8 3209 3430 3422 3422
q9 8645 8615 8675 8615
q10 4038 3788 3864 3788
q11 589 484 503 484
q12 865 675 664 664
q13 15927 2995 3080 2995
q14 323 270 269 269
q15 526 489 482 482
q16 477 434 427 427
q17 1791 1479 1464 1464
q18 8031 7792 7830 7792
q19 1773 1468 1497 1468
q20 3124 1850 1852 1850
q21 9340 4695 4901 4695
q22 638 597 582 582
Total cold run time: 77165 ms
Total hot run time: 55608 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2174944562
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2174993090
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2182958700
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2174968611
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2174998205
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2175411317
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) optimize out of range error message [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36450:
URL: https://github.com/apache/doris/pull/36450#issuecomment-2175009327
TeamCity be ut coverage result:
Function Coverage: 36.48% (9003/24679)
Line Coverage: 28.02% (73757/263196)
Region Coverage: 27.50% (38311/139321)
Branch Coverage: 24.19% (19520/80706)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0f1580b455a15b5346b22234ef92652e24390f56_0f1580b455a15b5346b22234ef92652e24390f56/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org