You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xinyiZzz (via GitHub)" <gi...@apache.org> on 2024/03/30 14:15:12 UTC
[PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
xinyiZzz opened a new pull request, #33071:
URL: https://github.com/apache/doris/pull/33071
## Proposed changes
Bug introduced by #32039
thread_context.h:292] __builtin_unreachable
```
10# doris::QueryThreadContext::init() at /home/zcp/repo_center/doris_master/doris/be/src/runtime/thread_context.h:306
11# doris::MemTable::MemTable(long, doris::TabletSchema const*, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const*, doris::TupleDescriptor*, bool, doris::PartialUpdateInfo*, std::shared_ptr<doris::MemTracker> const&, std::shared_ptr<doris::MemTracker> const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/memtable.cpp:69
12# doris::MemTableWriter::_reset_mem_table() at /home/zcp/repo_center/doris_master/doris/be/src/olap/memtable_writer.cpp:217
13# doris::MemTableWriter::flush_async() at /home/zcp/repo_center/doris_master/doris/be/src/olap/memtable_writer.cpp:171
14# doris::MemTableMemoryLimiter::_flush_memtable(std::weak_ptr<doris::MemTableWriter>, long) at /home/zcp/repo_center/doris_master/doris/be/src/olap/memtable_memory_limiter.cpp:193
15# doris::MemTableMemoryLimiter::_flush_active_memtables(long) at /home/zcp/repo_center/doris_master/doris/be/src/olap/memtable_memory_limiter.cpp:165
16# doris::MemTableMemoryLimiter::handle_memtable_flush() at /home/zcp/repo_center/doris_master/doris/be/src/olap/memtable_memory_limiter.cpp:146
```
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #33071:
URL: https://github.com/apache/doris/pull/33071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33071:
URL: https://github.com/apache/doris/pull/33071#issuecomment-2028097660
<details>
<summary>TPC-DS: <b>Total hot run time: 182676 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c61053764c0c88ccb5cf77bef4cbaa8c8718928d, data reload: false
query1 898 1127 1117 1117
query2 6298 1918 2002 1918
query3 6652 217 208 208
query4 24976 21528 21639 21528
query5 4154 400 382 382
query6 268 178 175 175
query7 4599 303 299 299
query8 229 189 173 173
query9 8479 2203 2196 2196
query10 448 260 263 260
query11 14904 14567 14478 14478
query12 149 94 98 94
query13 1633 383 397 383
query14 8433 6951 6910 6910
query15 204 176 182 176
query16 6821 278 287 278
query17 969 611 584 584
query18 1865 294 287 287
query19 213 163 169 163
query20 100 97 98 97
query21 201 135 131 131
query22 4993 4954 4918 4918
query23 33333 32584 32727 32584
query24 12900 3261 3214 3214
query25 733 464 422 422
query26 2041 177 174 174
query27 3300 385 385 385
query28 7102 1885 1869 1869
query29 1268 610 614 610
query30 309 160 166 160
query31 1069 765 751 751
query32 96 70 61 61
query33 758 257 256 256
query34 1448 504 503 503
query35 907 752 738 738
query36 993 872 868 868
query37 271 81 76 76
query38 3716 3638 3565 3565
query39 1662 1616 1598 1598
query40 263 154 151 151
query41 48 47 47 47
query42 113 108 108 108
query43 443 397 404 397
query44 1123 723 713 713
query45 296 264 282 264
query46 1104 841 834 834
query47 2031 1919 1947 1919
query48 391 335 318 318
query49 975 380 384 380
query50 841 422 408 408
query51 6873 6793 6689 6689
query52 111 102 109 102
query53 382 310 299 299
query54 299 255 245 245
query55 92 80 89 80
query56 247 234 230 230
query57 1276 1210 1186 1186
query58 244 236 226 226
query59 2711 2347 2167 2167
query60 263 241 236 236
query61 95 92 93 92
query62 646 437 457 437
query63 307 281 282 281
query64 5827 3351 3319 3319
query65 3041 3005 3024 3005
query66 1314 341 336 336
query67 15630 15015 14888 14888
query68 8713 576 581 576
query69 560 328 331 328
query70 1219 1122 1137 1122
query71 484 272 283 272
query72 6291 2615 2474 2474
query73 797 324 319 319
query74 6717 6409 6433 6409
query75 3309 2293 2255 2255
query76 4991 1088 1182 1088
query77 567 255 257 255
query78 10945 10152 10139 10139
query79 9824 545 546 545
query80 1562 440 418 418
query81 505 237 225 225
query82 620 105 100 100
query83 215 170 167 167
query84 259 88 91 88
query85 1380 302 295 295
query86 383 278 283 278
query87 3698 3486 3464 3464
query88 3799 2345 2330 2330
query89 557 369 373 369
query90 2031 184 184 184
query91 141 109 115 109
query92 69 52 54 52
query93 6928 541 533 533
query94 1296 205 201 201
query95 436 335 333 333
query96 615 274 272 272
query97 2667 2543 2485 2485
query98 239 219 214 214
query99 1266 831 859 831
Total cold run time: 299461 ms
Total hot run time: 182676 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #33071:
URL: https://github.com/apache/doris/pull/33071#issuecomment-2028083857
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33071:
URL: https://github.com/apache/doris/pull/33071#issuecomment-2028094735
<details>
<summary>TPC-H: <b>Total hot run time: 38673 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c61053764c0c88ccb5cf77bef4cbaa8c8718928d, data reload: false
------ Round 1 ----------------------------------
q1 17645 4152 4067 4067
q2 2115 196 180 180
q3 10472 1246 1413 1246
q4 10207 866 1045 866
q5 7467 2911 2913 2911
q6 214 131 130 130
q7 1089 646 615 615
q8 9416 2006 2037 2006
q9 6688 6216 6190 6190
q10 8472 3505 3509 3505
q11 449 240 235 235
q12 388 223 215 215
q13 17799 2885 2902 2885
q14 269 244 241 241
q15 525 486 496 486
q16 495 394 381 381
q17 947 918 919 918
q18 7447 6433 6368 6368
q19 1582 1527 1522 1522
q20 631 319 308 308
q21 3534 3095 3115 3095
q22 360 303 311 303
Total cold run time: 108211 ms
Total hot run time: 38673 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4112 4102 4126 4102
q2 339 229 230 229
q3 2985 2998 2996 2996
q4 1928 1869 1856 1856
q5 5277 5236 5203 5203
q6 212 128 127 127
q7 2239 1822 1810 1810
q8 3193 3257 3265 3257
q9 8451 8416 8399 8399
q10 3757 3985 4014 3985
q11 569 485 476 476
q12 764 598 582 582
q13 13696 3088 3069 3069
q14 321 275 306 275
q15 540 501 503 501
q16 507 460 481 460
q17 1773 1750 1745 1745
q18 8252 7649 7663 7649
q19 1974 1684 1665 1665
q20 2010 1846 1843 1843
q21 5067 5034 4981 4981
q22 509 428 416 416
Total cold run time: 68475 ms
Total hot run time: 55626 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33071:
URL: https://github.com/apache/doris/pull/33071#issuecomment-2028125589
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33071:
URL: https://github.com/apache/doris/pull/33071#issuecomment-2028085190
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33071:
URL: https://github.com/apache/doris/pull/33071#issuecomment-2028083840
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33071:
URL: https://github.com/apache/doris/pull/33071#issuecomment-2028125546
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33071:
URL: https://github.com/apache/doris/pull/33071#issuecomment-2028089016
TeamCity be ut coverage result:
Function Coverage: 35.55% (8840/24864)
Line Coverage: 27.29% (72488/265605)
Region Coverage: 26.50% (37522/141566)
Branch Coverage: 23.31% (19132/82060)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c61053764c0c88ccb5cf77bef4cbaa8c8718928d_c61053764c0c88ccb5cf77bef4cbaa8c8718928d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33071:
URL: https://github.com/apache/doris/pull/33071#issuecomment-2028099323
<details>
<summary>ClickBench: <b>Total hot run time: 29.61 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c61053764c0c88ccb5cf77bef4cbaa8c8718928d, data reload: false
query1 0.04 0.04 0.03
query2 0.07 0.04 0.04
query3 0.25 0.05 0.05
query4 1.67 0.07 0.07
query5 0.49 0.47 0.49
query6 1.14 0.65 0.65
query7 0.01 0.01 0.01
query8 0.06 0.05 0.05
query9 0.55 0.51 0.51
query10 0.56 0.56 0.57
query11 0.15 0.12 0.11
query12 0.14 0.12 0.12
query13 0.60 0.59 0.59
query14 0.77 0.78 0.79
query15 0.88 0.84 0.82
query16 0.36 0.36 0.36
query17 0.98 0.98 0.96
query18 0.24 0.25 0.27
query19 1.81 1.77 1.72
query20 0.01 0.02 0.02
query21 15.56 0.74 0.70
query22 3.31 5.36 1.19
query23 17.67 1.26 1.07
query24 1.46 0.21 0.23
query25 0.14 0.09 0.09
query26 0.29 0.17 0.19
query27 0.08 0.08 0.08
query28 13.83 0.99 0.96
query29 12.67 3.38 3.35
query30 0.28 0.09 0.07
query31 2.82 0.38 0.39
query32 3.30 0.47 0.48
query33 2.81 2.85 2.91
query34 15.51 4.39 4.38
query35 4.38 4.39 4.36
query36 0.67 0.48 0.47
query37 0.20 0.18 0.17
query38 0.18 0.15 0.16
query39 0.05 0.04 0.05
query40 0.18 0.16 0.14
query41 0.10 0.05 0.05
query42 0.07 0.05 0.05
query43 0.05 0.05 0.04
Total cold run time: 106.39 s
Total hot run time: 29.61 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Fix MemTableWriter flush_async attach task in thread context [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33071:
URL: https://github.com/apache/doris/pull/33071#issuecomment-2028101812
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit c61053764c0c88ccb5cf77bef4cbaa8c8718928d with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 15.4 seconds inserted 10000000 Rows, about 649K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org