You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "heguanhui (via GitHub)" <gi...@apache.org> on 2026/05/16 13:51:58 UTC
[PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
heguanhui opened a new pull request, #63310:
URL: https://github.com/apache/doris/pull/63310
## What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
`CatalogRecycleBin.runAfterCatalogReady()` captures `System.currentTimeMillis()` once at the beginning of the method and shares this timestamp across `erasePartition()`, `eraseTable()`, and `eraseDatabase()`. Since each erase operation can take significant time (I/O, log writes, lock acquisition), the later methods use a stale timestamp for expiration checks, causing delayed cleanup of tables and databases.
For example, if `erasePartition` takes 5 minutes, `eraseTable` and `eraseDatabase` will use a timestamp that is 5 minutes old, potentially skipping items that became eligible for cleanup during that period.
### Fix
Each erase method now gets its own fresh `System.currentTimeMillis()` call, ensuring accurate expiration checks.
**Before:**
```java
long currentTimeMs = System.currentTimeMillis();
erasePartition(currentTimeMs);
eraseTable(currentTimeMs);
eraseDatabase(currentTimeMs);
```
**After:**
```java
erasePartition(System.currentTimeMillis());
eraseTable(System.currentTimeMillis());
eraseDatabase(System.currentTimeMillis());
```
## Release note
Fix delayed cleanup of tables and databases in CatalogRecycleBin when partition erasure takes significant time
## Check List (For Author)
- Test: Unit Test
- Added `testEraseUsesFreshCurrentTime` in `CatalogRecycleBinTest` that creates a table with 1000 partitions, force-drops them, and verifies that tables and databases with expired recycle times are properly cleaned up even when partition erasure takes time
- Behavior changed: No (only fixes timing accuracy, no API or config changes)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4516488278
<details>
<summary>TPC-DS: <b>Total hot run time: 169884 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 98c43256eae708c582b6c51e66fe460dafae051a, data reload: false
query5 4310 654 536 536
query6 359 212 192 192
query7 4256 554 314 314
query8 328 238 208 208
query9 8824 4019 4025 4019
query10 442 364 298 298
query11 5776 2393 2207 2207
query12 181 128 125 125
query13 1347 617 415 415
query14 5917 5327 5047 5047
query14_1 4368 4343 4376 4343
query15 213 203 186 186
query16 1013 459 453 453
query17 1101 732 605 605
query18 2536 499 366 366
query19 224 208 170 170
query20 147 133 128 128
query21 218 141 119 119
query22 13771 13535 13375 13375
query23 17276 16365 16081 16081
query23_1 16290 16298 16137 16137
query24 7469 1747 1312 1312
query24_1 1297 1283 1319 1283
query25 574 504 433 433
query26 1330 319 171 171
query27 2679 543 329 329
query28 4460 1954 1970 1954
query29 1028 638 521 521
query30 311 237 202 202
query31 1122 1070 946 946
query32 92 76 77 76
query33 545 370 316 316
query34 1161 1130 635 635
query35 774 785 696 696
query36 1370 1346 1172 1172
query37 158 107 92 92
query38 3202 3175 3110 3110
query39 938 934 924 924
query39_1 870 885 887 885
query40 235 151 128 128
query41 72 70 68 68
query42 113 111 111 111
query43 328 336 289 289
query44
query45 223 205 199 199
query46 1059 1185 737 737
query47 2298 2323 2182 2182
query48 398 415 312 312
query49 739 520 406 406
query50 1008 353 245 245
query51 4407 4333 4249 4249
query52 108 108 98 98
query53 259 292 211 211
query54 328 291 279 279
query55 99 92 88 88
query56 331 332 315 315
query57 1401 1401 1303 1303
query58 314 315 270 270
query59 1565 1642 1413 1413
query60 315 344 309 309
query61 162 155 154 154
query62 668 626 553 553
query63 242 199 205 199
query64 2403 781 637 637
query65
query66 1733 486 347 347
query67 30355 29982 29734 29734
query68
query69 447 346 314 314
query70 980 1020 986 986
query71 306 277 272 272
query72 2984 2706 2405 2405
query73 836 751 453 453
query74 5060 4971 4744 4744
query75 2691 2597 2259 2259
query76 2270 1152 818 818
query77 405 403 336 336
query78 12276 12230 11577 11577
query79 1471 1082 734 734
query80 1237 535 458 458
query81 513 277 241 241
query82 1341 158 126 126
query83 355 287 243 243
query84 263 138 113 113
query85 930 573 444 444
query86 433 343 314 314
query87 3482 3345 3220 3220
query88 3513 2672 2688 2672
query89 447 385 333 333
query90 1802 179 190 179
query91 175 169 140 140
query92 82 78 74 74
query93 1445 1436 943 943
query94 629 360 303 303
query95 683 468 336 336
query96 998 759 342 342
query97 2712 2691 2562 2562
query98 245 226 222 222
query99 1132 1121 987 987
Total cold run time: 254513 ms
Total hot run time: 169884 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4603890244
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4741838679
# FE Regression Coverage Report
Increment line coverage ` 90.62% (58/64)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_31f5e7b8e267f8386f1547e9f8cca674c4a7765e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_31f5e7b8e267f8386f1547e9f8cca674c4a7765e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4731449022
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63310:
URL: https://github.com/apache/doris/pull/63310#discussion_r3425103907
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##########
@@ -67,6 +69,14 @@ public class CatalogRecycleBin extends MasterDaemon implements Writable {
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+ // Injectable clock source, defaults to system clock
+ private LongSupplier clock = System::currentTimeMillis;
Review Comment:
`CatalogRecycleBin.write()` still serializes `this` with `GsonUtils.GSON.toJson(this)` and `readFieldsWithGson()` immediately calls `GsonUtils.GSON.fromJson(..., CatalogRecycleBin.class)` on that JSON. Because this new test hook is not `transient` or otherwise excluded, the image JSON now contains a `LongSupplier` implementation. On image load Gson has to deserialize that interface field even though the returned object is ignored, which can fail and at minimum persists a non-deterministic test-only clock implementation. Please keep the injected clock out of the persisted image.
```suggestion
private transient LongSupplier clock = System::currentTimeMillis;
```
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##########
@@ -1393,13 +1403,14 @@ public void addTabletToInvertedIndex() {
@Override
protected void runAfterCatalogReady() {
- long currentTimeMs = System.currentTimeMillis();
// should follow the partition/table/db order
// in case of partition(table) is still in recycle bin but table(db) is missing
+ // Each erase method gets its own currentTimeMs to avoid using a stale timestamp,
+ // since previous erase operations may take significant time.
int keepNum = Config.max_same_name_catalog_trash_num;
- erasePartition(currentTimeMs, keepNum);
- eraseTable(currentTimeMs, keepNum);
- eraseDatabase(currentTimeMs, keepNum);
+ erasePartition(clock.getAsLong(), keepNum);
+ eraseTable(clock.getAsLong(), keepNum);
+ eraseDatabase(clock.getAsLong(), keepNum);
Review Comment:
This fresh timestamp can remove a parent after its child phase already skipped the child, which violates the invariant in the comment above. Concrete drop-database case: tables are recycled before the db in `InternalCatalog.unprotectDropDb()`/`recycleDatabase()`. If `eraseTable()` is called at `tableRecycleTime + expire - 1` it leaves the table in `idToTable`; if that phase then takes long enough, this later `eraseDatabase(clock.getAsLong(), ...)` can see the db as expired and the normal expired branch only removes `idToDatabase`/logs `OP_ERASE_DB` (it does not call `eraseAllTables`). After replay or image rebuild the table remains in the recycle bin with its database already erased, exactly what the partition/table/db ordering is supposed to prevent. The same boundary exists between partition and table. Please either use a consistent cutoff for dependent child/parent phases, or make parent erasure check/erase remaining children before logging the parent erase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4728462262
<details>
<summary>ClickBench: <b>Total hot run time: 25.23 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b977d0b4c19914042cee8e3f06c8358f43aa43a, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.14 0.13
query4 1.60 0.13 0.15
query5 0.25 0.22 0.22
query6 1.27 1.09 1.07
query7 0.04 0.01 0.01
query8 0.06 0.04 0.04
query9 0.37 0.31 0.30
query10 0.54 0.55 0.55
query11 0.22 0.14 0.14
query12 0.18 0.14 0.15
query13 0.47 0.47 0.48
query14 1.01 1.00 0.99
query15 0.61 0.59 0.59
query16 0.32 0.33 0.33
query17 1.06 1.10 1.11
query18 0.23 0.21 0.21
query19 2.02 1.93 2.03
query20 0.02 0.01 0.01
query21 15.45 0.22 0.14
query22 4.71 0.06 0.05
query23 16.13 0.32 0.13
query24 2.98 0.43 0.33
query25 0.11 0.05 0.04
query26 0.74 0.21 0.15
query27 0.04 0.03 0.04
query28 3.54 0.96 0.52
query29 12.47 4.29 3.45
query30 0.27 0.15 0.15
query31 2.77 0.60 0.31
query32 3.22 0.59 0.48
query33 3.24 3.19 3.33
query34 15.87 4.20 3.51
query35 3.47 3.54 3.56
query36 0.55 0.43 0.42
query37 0.09 0.07 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.18 0.17 0.16
query41 0.08 0.04 0.03
query42 0.03 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 96.71 s
Total hot run time: 25.23 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4728408466
<details>
<summary>TPC-DS: <b>Total hot run time: 175911 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b977d0b4c19914042cee8e3f06c8358f43aa43a, data reload: false
query5 4299 629 473 473
query6 435 192 173 173
query7 4816 561 308 308
query8 361 219 225 219
query9 8754 4164 4108 4108
query10 457 306 260 260
query11 5844 2369 2116 2116
query12 155 103 100 100
query13 1263 606 469 469
query14 6629 5405 5088 5088
query14_1 4381 4381 4392 4381
query15 208 200 183 183
query16 1052 458 451 451
query17 1141 738 600 600
query18 2739 517 357 357
query19 258 187 149 149
query20 114 109 105 105
query21 219 144 129 129
query22 13649 13684 13386 13386
query23 17523 16670 16208 16208
query23_1 16256 16308 16295 16295
query24 7488 1815 1305 1305
query24_1 1338 1331 1343 1331
query25 579 460 406 406
query26 1293 314 176 176
query27 2575 557 358 358
query28 4351 2102 2065 2065
query29 1085 642 511 511
query30 311 237 207 207
query31 1136 1088 964 964
query32 102 61 62 61
query33 532 328 254 254
query34 1197 1212 660 660
query35 745 816 664 664
query36 1370 1388 1238 1238
query37 148 108 89 89
query38 3221 3134 3055 3055
query39 917 926 894 894
query39_1 899 853 869 853
query40 221 118 100 100
query41 63 62 62 62
query42 93 93 97 93
query43 328 328 287 287
query44 1441 780 793 780
query45 194 189 176 176
query46 1060 1232 764 764
query47 2343 2386 2195 2195
query48 400 385 309 309
query49 634 460 346 346
query50 969 364 254 254
query51 4367 4341 4287 4287
query52 88 89 77 77
query53 255 259 186 186
query54 265 216 204 204
query55 81 74 69 69
query56 242 224 204 204
query57 1437 1437 1337 1337
query58 231 206 206 206
query59 1558 1676 1442 1442
query60 288 241 226 226
query61 157 153 153 153
query62 684 644 583 583
query63 236 189 194 189
query64 2458 765 595 595
query65 4849 4730 4751 4730
query66 1711 506 336 336
query67 29747 29683 29541 29541
query68 3290 1572 999 999
query69 421 296 292 292
query70 1038 976 951 951
query71 295 236 218 218
query72 2949 2658 2414 2414
query73 846 806 425 425
query74 5094 5002 4765 4765
query75 2638 2584 2253 2253
query76 2369 1209 807 807
query77 356 375 279 279
query78 12415 12563 11854 11854
query79 1341 1244 767 767
query80 616 483 374 374
query81 460 282 241 241
query82 604 160 124 124
query83 346 276 247 247
query84 263 147 113 113
query85 867 519 416 416
query86 357 309 263 263
query87 3370 3327 3138 3138
query88 3700 2800 2775 2775
query89 423 393 328 328
query90 1973 183 176 176
query91 176 165 136 136
query92 72 61 57 57
query93 1484 1440 939 939
query94 550 354 322 322
query95 680 489 346 346
query96 1138 811 338 338
query97 2715 2706 2547 2547
query98 212 210 202 202
query99 1176 1159 1031 1031
Total cold run time: 260736 ms
Total hot run time: 175911 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4709321374
<details>
<summary>TPC-DS: <b>Total hot run time: 168866 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a15360e85ac725af3a16308e33e1996bd355f920, data reload: false
query5 4332 627 476 476
query6 429 192 175 175
query7 4860 568 312 312
query8 369 223 199 199
query9 8774 4040 4043 4040
query10 441 312 260 260
query11 5941 2338 2183 2183
query12 162 102 99 99
query13 1265 607 418 418
query14 6386 5429 5083 5083
query14_1 4404 4436 4388 4388
query15 205 198 175 175
query16 1000 456 421 421
query17 1121 721 584 584
query18 2655 494 359 359
query19 202 196 149 149
query20 113 111 109 109
query21 224 150 121 121
query22 13634 13585 13366 13366
query23 17418 16501 16233 16233
query23_1 16195 16228 16295 16228
query24 7480 1772 1266 1266
query24_1 1314 1314 1317 1314
query25 581 468 399 399
query26 1298 310 178 178
query27 2675 570 332 332
query28 4463 2017 2041 2017
query29 1081 646 501 501
query30 313 231 202 202
query31 1105 1075 965 965
query32 111 66 60 60
query33 543 328 261 261
query34 1178 1132 669 669
query35 761 811 681 681
query36 1459 1359 1256 1256
query37 160 102 87 87
query38 3193 3131 3030 3030
query39 913 909 908 908
query39_1 867 873 874 873
query40 227 118 100 100
query41 63 73 59 59
query42 92 92 98 92
query43 315 320 272 272
query44
query45 200 185 176 176
query46 1061 1230 760 760
query47 2307 2338 2180 2180
query48 366 421 290 290
query49 619 490 335 335
query50 1020 350 257 257
query51 4377 4385 4212 4212
query52 86 86 76 76
query53 255 266 193 193
query54 262 216 217 216
query55 80 74 70 70
query56 227 225 222 222
query57 1433 1413 1334 1334
query58 234 209 201 201
query59 1599 1639 1377 1377
query60 285 244 230 230
query61 158 152 157 152
query62 702 660 589 589
query63 229 184 188 184
query64 2493 763 616 616
query65
query66 1743 451 340 340
query67 29763 29657 28924 28924
query68
query69 412 306 271 271
query70 992 933 942 933
query71 297 222 208 208
query72 2987 2606 2333 2333
query73 836 766 454 454
query74 5122 4943 4792 4792
query75 2650 2581 2267 2267
query76 2294 1136 776 776
query77 352 367 289 289
query78 12517 12563 12023 12023
query79 1435 1089 772 772
query80 921 465 388 388
query81 501 288 247 247
query82 583 158 118 118
query83 325 273 251 251
query84
query85 882 537 424 424
query86 400 303 298 298
query87 3402 3412 3193 3193
query88 3602 2743 2733 2733
query89 417 387 327 327
query90 1867 189 178 178
query91 173 156 130 130
query92 67 62 56 56
query93 1434 1490 862 862
query94 628 357 319 319
query95 680 408 445 408
query96 1099 803 328 328
query97 2708 2671 2622 2622
query98 208 204 205 204
query99 1117 1172 1004 1004
Total cold run time: 251091 ms
Total hot run time: 168866 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4710168037
# FE UT Coverage Report
Increment line coverage ` 100.00% (9/9)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_a15360e85ac725af3a16308e33e1996bd355f920/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_a15360e85ac725af3a16308e33e1996bd355f920/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4467046977
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4745016711
<details>
<summary>ClickBench: <b>Total hot run time: 25.41 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 31f5e7b8e267f8386f1547e9f8cca674c4a7765e, data reload: false
query1 0.01 0.00 0.01
query2 0.10 0.05 0.05
query3 0.26 0.14 0.14
query4 1.61 0.13 0.13
query5 0.25 0.23 0.22
query6 1.30 1.07 1.05
query7 0.04 0.01 0.00
query8 0.10 0.03 0.03
query9 0.38 0.33 0.32
query10 0.58 0.58 0.58
query11 0.20 0.14 0.14
query12 0.18 0.16 0.14
query13 0.48 0.48 0.49
query14 1.03 0.99 1.01
query15 0.65 0.61 0.62
query16 0.34 0.35 0.32
query17 1.13 1.15 1.15
query18 0.23 0.22 0.22
query19 2.00 1.91 1.92
query20 0.01 0.01 0.02
query21 15.44 0.22 0.15
query22 4.81 0.05 0.06
query23 16.12 0.30 0.12
query24 3.04 0.47 0.27
query25 0.10 0.06 0.05
query26 0.73 0.20 0.16
query27 0.05 0.04 0.03
query28 3.43 0.93 0.57
query29 12.51 4.31 3.47
query30 0.27 0.15 0.15
query31 2.78 0.60 0.32
query32 3.22 0.60 0.50
query33 3.14 3.23 3.22
query34 15.62 4.28 3.56
query35 3.55 3.53 3.53
query36 0.56 0.43 0.42
query37 0.10 0.06 0.07
query38 0.05 0.04 0.03
query39 0.04 0.02 0.02
query40 0.19 0.16 0.15
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 96.8 s
Total hot run time: 25.41 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4740173841
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4709789165
<details>
<summary>TPC-DS: <b>Total hot run time: 168965 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a15360e85ac725af3a16308e33e1996bd355f920, data reload: false
query5 4305 631 470 470
query6 451 191 171 171
query7 4880 572 298 298
query8 357 209 191 191
query9 8742 3980 3977 3977
query10 442 322 253 253
query11 5852 2307 2175 2175
query12 157 100 95 95
query13 1280 595 430 430
query14 6385 5377 5069 5069
query14_1 4365 4372 4345 4345
query15 205 202 178 178
query16 1025 446 439 439
query17 1132 713 582 582
query18 2461 477 362 362
query19 210 190 144 144
query20 113 107 106 106
query21 225 145 118 118
query22 13576 13482 13449 13449
query23 17231 16498 16067 16067
query23_1 16191 16363 16273 16273
query24 7581 1752 1317 1317
query24_1 1326 1314 1320 1314
query25 594 450 402 402
query26 1307 312 168 168
query27 2723 531 339 339
query28 4438 2025 2017 2017
query29 1085 625 490 490
query30 308 236 195 195
query31 1114 1081 946 946
query32 109 62 60 60
query33 522 317 261 261
query34 1208 1130 669 669
query35 787 785 683 683
query36 1423 1398 1283 1283
query37 154 98 86 86
query38 3251 3184 3116 3116
query39 963 972 968 968
query39_1 936 914 910 910
query40 224 121 101 101
query41 65 61 61 61
query42 95 92 91 91
query43 318 323 280 280
query44
query45 200 183 178 178
query46 1123 1223 709 709
query47 2398 2371 2271 2271
query48 391 425 291 291
query49 614 458 331 331
query50 942 335 260 260
query51 4327 4353 4304 4304
query52 86 86 74 74
query53 242 263 193 193
query54 270 218 215 215
query55 79 77 69 69
query56 216 214 213 213
query57 1449 1402 1298 1298
query58 249 203 212 203
query59 1544 1656 1420 1420
query60 286 240 225 225
query61 154 149 141 141
query62 695 659 594 594
query63 226 180 188 180
query64 2536 738 581 581
query65
query66 1799 458 329 329
query67 29847 29677 29595 29595
query68
query69 434 303 266 266
query70 950 963 910 910
query71 307 218 205 205
query72 2929 2640 2350 2350
query73 877 820 429 429
query74 5127 4975 4751 4751
query75 2664 2562 2229 2229
query76 2340 1130 798 798
query77 356 379 294 294
query78 12388 12380 11858 11858
query79 1467 1009 756 756
query80 1270 449 366 366
query81 518 272 237 237
query82 720 156 120 120
query83 349 284 240 240
query84
query85 888 491 407 407
query86 435 305 291 291
query87 3385 3398 3165 3165
query88 3650 2745 2750 2745
query89 443 373 322 322
query90 1926 175 166 166
query91 201 156 134 134
query92 66 62 56 56
query93 1586 1512 846 846
query94 753 320 310 310
query95 679 370 420 370
query96 1031 755 333 333
query97 2678 2683 2581 2581
query98 210 220 199 199
query99 1132 1180 1036 1036
Total cold run time: 251914 ms
Total hot run time: 168965 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "yx-keith (via GitHub)" <gi...@apache.org>.
yx-keith commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4737275285
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4514248107
rub buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4698777246
<details>
<summary>TPC-DS: <b>Total hot run time: 168029 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 621a300298116963c26e950558d255700bcfca15, data reload: false
query5 4318 613 480 480
query6 442 190 169 169
query7 4845 552 316 316
query8 364 215 203 203
query9 8771 3974 3975 3974
query10 428 310 251 251
query11 5942 2352 2162 2162
query12 152 102 96 96
query13 1245 650 428 428
query14 6385 5305 5014 5014
query14_1 4336 4324 4314 4314
query15 198 198 175 175
query16 1020 432 414 414
query17 1083 672 539 539
query18 2526 450 335 335
query19 189 178 133 133
query20 114 107 106 106
query21 216 138 116 116
query22 13667 13519 13399 13399
query23 17164 16424 16062 16062
query23_1 16198 16260 16222 16222
query24 7568 1787 1294 1294
query24_1 1315 1286 1315 1286
query25 552 468 402 402
query26 1295 311 162 162
query27 2686 543 346 346
query28 4501 1999 2004 1999
query29 1079 612 489 489
query30 310 237 198 198
query31 1102 1072 939 939
query32 108 61 59 59
query33 553 328 252 252
query34 1187 1142 648 648
query35 752 793 676 676
query36 1385 1409 1244 1244
query37 157 109 100 100
query38 3206 3115 3040 3040
query39 926 920 912 912
query39_1 895 881 894 881
query40 222 126 105 105
query41 72 64 68 64
query42 95 96 101 96
query43 314 321 282 282
query44
query45 198 189 191 189
query46 1088 1201 713 713
query47 2383 2399 2246 2246
query48 393 415 300 300
query49 640 470 360 360
query50 997 350 267 267
query51 4309 4262 4228 4228
query52 92 90 78 78
query53 248 264 193 193
query54 279 226 206 206
query55 78 80 73 73
query56 247 229 231 229
query57 1424 1409 1312 1312
query58 286 217 216 216
query59 1556 1626 1458 1458
query60 290 260 232 232
query61 178 203 149 149
query62 705 645 566 566
query63 225 182 182 182
query64 2507 764 615 615
query65
query66 1757 454 338 338
query67 29756 29620 29464 29464
query68
query69 440 302 257 257
query70 954 944 955 944
query71 288 219 206 206
query72 2774 2612 2346 2346
query73 856 790 433 433
query74 5088 4938 4790 4790
query75 2616 2556 2218 2218
query76 2332 1151 762 762
query77 345 374 282 282
query78 12380 12474 11754 11754
query79 1404 1072 770 770
query80 1286 465 377 377
query81 516 275 245 245
query82 598 154 121 121
query83 344 276 244 244
query84
query85 884 506 406 406
query86 432 297 279 279
query87 3401 3325 3209 3209
query88 3637 2735 2736 2735
query89 424 386 324 324
query90 1969 185 175 175
query91 166 158 136 136
query92 62 62 56 56
query93 1577 1482 889 889
query94 724 357 295 295
query95 672 458 334 334
query96 1140 793 341 341
query97 2721 2676 2563 2563
query98 207 199 201 199
query99 1155 1165 1031 1031
Total cold run time: 251374 ms
Total hot run time: 168029 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4709696881
<details>
<summary>TPC-H: <b>Total hot run time: 28297 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a15360e85ac725af3a16308e33e1996bd355f920, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17696 3981 3931 3931
q2 q3 10842 1361 780 780
q4 4682 472 339 339
q5 7563 851 596 596
q6 179 165 135 135
q7 782 814 635 635
q8 9815 1569 1464 1464
q9 7012 4434 4450 4434
q10 6805 1813 1498 1498
q11 432 268 243 243
q12 642 421 289 289
q13 18158 3656 2772 2772
q14 268 258 240 240
q15 q16 825 781 700 700
q17 1180 946 824 824
q18 6930 5781 5505 5505
q19 1252 1293 1078 1078
q20 495 409 253 253
q21 5769 2668 2281 2281
q22 432 357 300 300
Total cold run time: 101759 ms
Total hot run time: 28297 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4329 4211 4231 4211
q2 q3 4474 4941 4374 4374
q4 2065 2138 1386 1386
q5 4386 4280 4278 4278
q6 226 172 125 125
q7 1715 1606 1861 1606
q8 2530 2189 2107 2107
q9 7771 7810 7749 7749
q10 4773 4775 4324 4324
q11 580 444 367 367
q12 749 749 550 550
q13 3456 3525 2895 2895
q14 300 301 266 266
q15 q16 764 759 663 663
q17 1323 1321 1296 1296
q18 8076 7314 7060 7060
q19 1187 1099 1101 1099
q20 2207 2225 1935 1935
q21 5218 4545 4383 4383
q22 493 476 413 413
Total cold run time: 56622 ms
Total hot run time: 51087 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4743408390
# FE Regression Coverage Report
Increment line coverage ` 90.62% (58/64)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_31f5e7b8e267f8386f1547e9f8cca674c4a7765e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_31f5e7b8e267f8386f1547e9f8cca674c4a7765e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4744888390
<details>
<summary>TPC-H: <b>Total hot run time: 29377 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 31f5e7b8e267f8386f1547e9f8cca674c4a7765e, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17593 4003 4015 4003
q2 2042 322 194 194
q3 10325 1423 816 816
q4 4686 472 337 337
q5 7512 860 582 582
q6 185 170 141 141
q7 789 838 625 625
q8 9355 1615 1592 1592
q9 5828 4527 4494 4494
q10 6773 1788 1563 1563
q11 438 282 245 245
q12 627 422 289 289
q13 18126 3359 2756 2756
q14 266 261 248 248
q15 q16 783 786 723 723
q17 1040 964 887 887
q18 7087 5748 5678 5678
q19 1303 1623 1074 1074
q20 495 400 270 270
q21 5924 2644 2561 2561
q22 438 360 299 299
Total cold run time: 101615 ms
Total hot run time: 29377 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4451 4292 4266 4266
q2 354 360 235 235
q3 4558 4978 4392 4392
q4 2103 2162 1383 1383
q5 4470 4311 4316 4311
q6 234 176 130 130
q7 1715 1805 1915 1805
q8 2648 2219 2323 2219
q9 8136 8596 7969 7969
q10 4835 4767 4351 4351
q11 586 414 388 388
q12 740 755 550 550
q13 3242 3745 2928 2928
q14 295 320 289 289
q15 q16 712 719 638 638
q17 1344 1316 1327 1316
q18 8055 7427 7238 7238
q19 1167 1131 1093 1093
q20 2216 2224 1938 1938
q21 5326 4757 4497 4497
q22 494 478 392 392
Total cold run time: 57681 ms
Total hot run time: 52328 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4698750263
<details>
<summary>TPC-H: <b>Total hot run time: 29041 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 621a300298116963c26e950558d255700bcfca15, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17696 4037 3949 3949
q2 q3 10758 1353 802 802
q4 4679 477 345 345
q5 7513 863 567 567
q6 175 168 136 136
q7 766 820 630 630
q8 9365 1468 1676 1468
q9 5835 4522 4476 4476
q10 6782 1743 1531 1531
q11 440 263 246 246
q12 631 428 296 296
q13 18117 3417 2789 2789
q14 264 261 236 236
q15 q16 821 771 702 702
q17 989 965 941 941
q18 7001 5647 5500 5500
q19 1321 1247 1106 1106
q20 535 390 263 263
q21 6229 2796 2741 2741
q22 446 387 317 317
Total cold run time: 100363 ms
Total hot run time: 29041 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5078 4588 4715 4588
q2 q3 4852 5160 4681 4681
q4 2110 2192 1358 1358
q5 4820 4765 4643 4643
q6 239 179 133 133
q7 1836 1711 1570 1570
q8 2397 2086 2069 2069
q9 7943 7615 7379 7379
q10 4701 4652 4217 4217
q11 522 378 347 347
q12 726 752 532 532
q13 3027 3355 2755 2755
q14 278 288 248 248
q15 q16 673 693 607 607
q17 1270 1246 1234 1234
q18 7327 6732 6797 6732
q19 1115 1063 1097 1063
q20 2214 2210 1937 1937
q21 5614 4483 4309 4309
q22 532 465 406 406
Total cold run time: 57274 ms
Total hot run time: 50808 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4708521917
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4740525076
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "yx-keith (via GitHub)" <gi...@apache.org>.
yx-keith commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4743352596
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4729711949
# FE Regression Coverage Report
Increment line coverage ` 88.06% (59/67)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_4b977d0b4c19914042cee8e3f06c8358f43aa43a_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_4b977d0b4c19914042cee8e3f06c8358f43aa43a_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on code in PR #63310:
URL: https://github.com/apache/doris/pull/63310#discussion_r3432537350
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##########
@@ -467,6 +479,7 @@ private void eraseTable(long currentTimeMs, int keepNum) {
if (tableInfo == null) {
continue;
}
+ erasePartitionsForTable(tableId);
Review Comment:
I already fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4710849762
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4716537274
<details>
<summary>TPC-DS: <b>Total hot run time: 175377 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c777b85bc8a8cb928dba405a4b684eebd819e8b9, data reload: false
query5 4324 625 464 464
query6 432 204 176 176
query7 4862 557 313 313
query8 372 213 195 195
query9 8762 4027 4053 4027
query10 440 308 253 253
query11 5911 2338 2165 2165
query12 156 99 99 99
query13 1265 594 452 452
query14 6394 5384 5056 5056
query14_1 4365 4405 4364 4364
query15 204 195 176 176
query16 1019 460 475 460
query17 1091 697 558 558
query18 2500 484 334 334
query19 195 184 134 134
query20 111 108 102 102
query21 213 135 118 118
query22 13734 13718 13335 13335
query23 17343 16579 16067 16067
query23_1 16291 16269 16324 16269
query24 7430 1770 1317 1317
query24_1 1309 1314 1311 1311
query25 528 435 366 366
query26 1282 314 166 166
query27 2682 551 338 338
query28 4449 2045 2020 2020
query29 1049 624 494 494
query30 312 237 197 197
query31 1118 1075 961 961
query32 104 63 58 58
query33 504 309 236 236
query34 1195 1174 670 670
query35 768 776 678 678
query36 1355 1415 1195 1195
query37 155 105 90 90
query38 3213 3144 3047 3047
query39 907 907 901 901
query39_1 886 880 876 876
query40 225 124 104 104
query41 72 68 67 67
query42 97 100 100 100
query43 326 329 286 286
query44 1444 785 785 785
query45 195 186 174 174
query46 1090 1214 775 775
query47 2369 2350 2209 2209
query48 414 421 298 298
query49 696 483 344 344
query50 971 355 261 261
query51 4344 4345 4278 4278
query52 87 88 76 76
query53 248 280 186 186
query54 284 210 184 184
query55 76 74 67 67
query56 241 218 216 216
query57 1433 1414 1309 1309
query58 232 214 208 208
query59 1553 1650 1486 1486
query60 285 249 222 222
query61 149 151 148 148
query62 686 652 582 582
query63 231 190 193 190
query64 2509 767 616 616
query65 4855 4823 4792 4792
query66 1776 460 340 340
query67 30059 29653 29526 29526
query68 3152 1512 916 916
query69 399 317 261 261
query70 1094 979 963 963
query71 295 233 214 214
query72 2906 2743 2466 2466
query73 843 775 454 454
query74 5151 4960 4764 4764
query75 2632 2610 2258 2258
query76 2340 1180 770 770
query77 384 377 280 280
query78 12412 12458 11824 11824
query79 1178 1179 741 741
query80 521 469 381 381
query81 448 278 241 241
query82 240 159 125 125
query83 269 278 251 251
query84 268 144 115 115
query85 824 516 418 418
query86 330 306 274 274
query87 3345 3351 3218 3218
query88 3717 2797 2812 2797
query89 407 385 329 329
query90 2147 186 181 181
query91 172 163 132 132
query92 66 59 56 56
query93 1427 1624 993 993
query94 538 349 306 306
query95 677 371 434 371
query96 1053 838 371 371
query97 2720 2709 2575 2575
query98 213 207 199 199
query99 1160 1156 1028 1028
Total cold run time: 259902 ms
Total hot run time: 175377 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4741917230
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4699132947
# FE Regression Coverage Report
Increment line coverage ` 66.67% (4/6)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_621a300298116963c26e950558d255700bcfca15_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_621a300298116963c26e950558d255700bcfca15_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4515788817
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4516659948
# FE UT Coverage Report
Increment line coverage ` 100.00% (3/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_98c43256eae708c582b6c51e66fe460dafae051a/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_98c43256eae708c582b6c51e66fe460dafae051a/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4517358263
# FE Regression Coverage Report
Increment line coverage ` 8.82% (3/34)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_98c43256eae708c582b6c51e66fe460dafae051a_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_98c43256eae708c582b6c51e66fe460dafae051a_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4709208412
<details>
<summary>TPC-H: <b>Total hot run time: 28825 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a15360e85ac725af3a16308e33e1996bd355f920, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17825 4106 4039 4039
q2 q3 10757 1401 815 815
q4 4724 477 355 355
q5 7878 894 567 567
q6 209 169 140 140
q7 827 842 617 617
q8 10433 1626 1628 1626
q9 6822 4485 4502 4485
q10 6806 1831 1504 1504
q11 436 263 247 247
q12 676 426 295 295
q13 18177 3345 2788 2788
q14 263 252 236 236
q15 q16 822 765 705 705
q17 1000 884 1027 884
q18 7154 5779 5536 5536
q19 1221 1307 1111 1111
q20 509 408 265 265
q21 5706 2573 2313 2313
q22 449 350 297 297
Total cold run time: 102694 ms
Total hot run time: 28825 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4348 4249 4273 4249
q2 q3 4555 4959 4301 4301
q4 2106 2226 1381 1381
q5 4487 4323 4304 4304
q6 251 176 129 129
q7 1717 2128 1590 1590
q8 2491 2232 2072 2072
q9 7924 7884 7899 7884
q10 4813 4782 4332 4332
q11 576 432 542 432
q12 757 748 556 556
q13 3222 3715 2997 2997
q14 303 324 287 287
q15 q16 716 753 645 645
q17 1352 1334 1315 1315
q18 7788 7242 6754 6754
q19 1098 1063 1101 1063
q20 2217 2204 1973 1973
q21 5257 5036 4458 4458
q22 542 486 399 399
Total cold run time: 56520 ms
Total hot run time: 51121 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4725000871
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on code in PR #63310:
URL: https://github.com/apache/doris/pull/63310#discussion_r3426559995
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##########
@@ -67,6 +69,14 @@ public class CatalogRecycleBin extends MasterDaemon implements Writable {
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+ // Injectable clock source, defaults to system clock
+ private LongSupplier clock = System::currentTimeMillis;
Review Comment:
I will fix this issue.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##########
@@ -1393,13 +1403,14 @@ public void addTabletToInvertedIndex() {
@Override
protected void runAfterCatalogReady() {
- long currentTimeMs = System.currentTimeMillis();
// should follow the partition/table/db order
// in case of partition(table) is still in recycle bin but table(db) is missing
+ // Each erase method gets its own currentTimeMs to avoid using a stale timestamp,
+ // since previous erase operations may take significant time.
int keepNum = Config.max_same_name_catalog_trash_num;
- erasePartition(currentTimeMs, keepNum);
- eraseTable(currentTimeMs, keepNum);
- eraseDatabase(currentTimeMs, keepNum);
+ erasePartition(clock.getAsLong(), keepNum);
+ eraseTable(clock.getAsLong(), keepNum);
+ eraseDatabase(clock.getAsLong(), keepNum);
Review Comment:
I will fix this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4728041328
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4743269091
# FE Regression Coverage Report
Increment line coverage ` 90.62% (58/64)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_31f5e7b8e267f8386f1547e9f8cca674c4a7765e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_31f5e7b8e267f8386f1547e9f8cca674c4a7765e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4733602631
# FE Regression Coverage Report
Increment line coverage ` 89.71% (61/68)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_c92c60e6d644d9e325f2adc1fcf690eae9e6b7eb_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_c92c60e6d644d9e325f2adc1fcf690eae9e6b7eb_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4698516603
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4610750129
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4708522903
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4717858446
# FE Regression Coverage Report
Increment line coverage ` 77.78% (7/9)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_c777b85bc8a8cb928dba405a4b684eebd819e8b9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_c777b85bc8a8cb928dba405a4b684eebd819e8b9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4728042061
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4730015690
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4743142686
# FE Regression Coverage Report
Increment line coverage ` 90.62% (58/64)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_31f5e7b8e267f8386f1547e9f8cca674c4a7765e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_31f5e7b8e267f8386f1547e9f8cca674c4a7765e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63310:
URL: https://github.com/apache/doris/pull/63310#discussion_r3427248747
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##########
@@ -574,7 +587,128 @@ public void replayEraseTable(long tableId) {
}
}
- private void erasePartition(long currentTimeMs, int keepNum) {
+ /**
+ * Erase all partitions belonging to the given table.
+ * This handles partitions that were not expired when erasePartition() collected
+ * its expired IDs, but became expired later due to time advancement during processing.
+ *
+ * <p>This method acquires its own locks with fine granularity:
+ * <ul>
+ * <li>Read lock to collect partition IDs</li>
+ * <li>Write lock for each individual partition cleanup</li>
+ * </ul>
+ *
+ * @param tableId the table ID whose partitions need to be erased
+ */
+ private void erasePartitionsForTable(long tableId) {
+ // 1. Collect orphan partition IDs under read lock (fast, non-blocking)
+ List<Long> partitionIds = new ArrayList<>();
+ readLock();
+ try {
+ for (Map.Entry<Long, RecyclePartitionInfo> entry : idToPartition.entrySet()) {
+ if (entry.getValue().getTableId() == tableId) {
+ partitionIds.add(entry.getKey());
+ }
+ }
+ } finally {
+ readUnlock();
+ }
+
+ // 2. Clean each orphan partition with individual write lock (fine granularity)
+ for (Long partitionId : partitionIds) {
+ writeLock();
+ try {
+ RecyclePartitionInfo pInfo = idToPartition.remove(partitionId);
+ if (pInfo == null) {
+ continue;
+ }
+ Partition partition = pInfo.getPartition();
+ Env.getCurrentEnv().onErasePartition(partition);
+ idToRecycleTime.remove(partitionId);
+
+ dbTblIdPartitionNameToIds.computeIfPresent(
+ Pair.of(pInfo.getDbId(), pInfo.getTableId()), (pair, partitionMap) -> {
+ partitionMap.computeIfPresent(partition.getName(), (name, idSet) -> {
+ idSet.remove(partitionId);
+ return idSet.isEmpty() ? null : idSet;
+ });
+ return partitionMap.isEmpty() ? null : partitionMap;
+ });
+
+ Env.getCurrentEnv().getEditLog().logErasePartition(partitionId);
+ LOG.info("erase orphan partition[{}] when erasing table[{}]", partitionId, tableId);
+ } finally {
+ writeUnlock();
+ }
+ }
+ }
+
+ /**
+ * Erase all tables belonging to the given database.
+ * This handles tables that were not expired when eraseTable() collected
+ * its expired IDs, but became expired later due to time advancement during processing.
+ *
+ * <p>This method acquires its own locks with fine granularity:
+ * <ul>
+ * <li>Read lock to collect table IDs</li>
+ * <li>For each table, erase its partitions then erase the table itself</li>
+ * </ul>
+ *
+ * @param dbInfo the RecycleDatabaseInfo containing the database and its table metadata
+ */
+ private void eraseTablesForDatabase(RecycleDatabaseInfo dbInfo) {
+ long dbId = dbInfo.getDb().getId();
+ Set<String> tableNames = Sets.newHashSet(dbInfo.getTableNames());
+ Set<Long> tableIdSet = Sets.newHashSet(dbInfo.getTableIds());
+
+ // 1. Collect orphan table IDs under read lock
+ List<Long> tableIds = new ArrayList<>();
+ readLock();
+ try {
+ for (Map.Entry<Long, RecycleTableInfo> entry : idToTable.entrySet()) {
+ RecycleTableInfo tableInfo = entry.getValue();
+ if (tableInfo.getDbId() == dbId
+ && tableNames.contains(tableInfo.getTable().getName())
Review Comment:
This still skips recycled tables from the same database if they were dropped before the database drop. Concrete sequence: drop table `t_old` in db `d`, then later drop database `d`. `RecycleDatabaseInfo` is built from `db.getTables()` at the database drop, so it does not contain `t_old`. When the db expires, this helper skips `t_old`, then `eraseDatabase()` removes/logs the db, leaving `idToTable` with a dbId that is no longer in the catalog or `idToDatabase`. That table is no longer recoverable and will be rebuilt from image as a recycle-bin table whose database is missing. For database parent erasure, please erase all recycled tables/partitions with this `dbId` before logging `OP_ERASE_DB`, or otherwise explicitly clean the remaining children by dbId. This is distinct from the existing thread's current-table drop-db case because it covers tables that were already recycled before the database drop and therefore are not in `dbInfo.getTableIds()`.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##########
@@ -467,6 +479,7 @@ private void eraseTable(long currentTimeMs, int keepNum) {
if (tableInfo == null) {
continue;
}
+ erasePartitionsForTable(tableId);
Review Comment:
This fixes the normal expired-table path, but the parallel `eraseTableWithSameName()` path still removes a recycled table without calling `erasePartitionsForTable`. A concrete case is: drop partition `p` from table `t`, then drop multiple tables named `t` in the same db so `max_same_name_catalog_trash_num` evicts the older table after min latency. `erasePartition()` can leave `p` because it is not expired and not over the partition keep count, then `eraseTableWithSameName()` removes/logs the table and leaves `p` with a missing parent table. Please apply the same partition cascade before removing/logging the table in the same-name path, and add a test for that path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4744983032
<details>
<summary>TPC-DS: <b>Total hot run time: 175566 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 31f5e7b8e267f8386f1547e9f8cca674c4a7765e, data reload: false
query5 4301 629 476 476
query6 447 195 180 180
query7 4817 569 316 316
query8 373 218 215 215
query9 8772 4072 4101 4072
query10 451 319 268 268
query11 5913 2367 2130 2130
query12 160 102 97 97
query13 1261 610 448 448
query14 6373 5403 5069 5069
query14_1 4425 4382 4362 4362
query15 206 199 175 175
query16 978 466 459 459
query17 942 708 583 583
query18 2428 485 377 377
query19 207 196 147 147
query20 113 115 108 108
query21 222 136 119 119
query22 13634 13576 13396 13396
query23 17476 16624 16124 16124
query23_1 16281 16227 16286 16227
query24 7400 1810 1294 1294
query24_1 1318 1313 1318 1313
query25 588 474 407 407
query26 1304 307 170 170
query27 2711 566 341 341
query28 4491 2038 2006 2006
query29 1135 595 476 476
query30 312 235 198 198
query31 1103 1061 949 949
query32 130 60 57 57
query33 522 317 247 247
query34 1183 1127 692 692
query35 811 773 671 671
query36 1413 1439 1249 1249
query37 157 113 90 90
query38 3200 3125 3045 3045
query39 941 920 890 890
query39_1 875 899 871 871
query40 215 119 98 98
query41 64 64 59 59
query42 91 96 95 95
query43 322 328 273 273
query44 1424 772 775 772
query45 194 182 173 173
query46 1049 1242 724 724
query47 2323 2385 2232 2232
query48 408 397 300 300
query49 620 467 336 336
query50 1064 351 255 255
query51 4405 4370 4322 4322
query52 87 86 75 75
query53 253 268 195 195
query54 265 227 189 189
query55 79 73 69 69
query56 234 221 215 215
query57 1430 1430 1330 1330
query58 246 214 200 200
query59 1600 1647 1471 1471
query60 286 247 227 227
query61 155 142 143 142
query62 704 651 587 587
query63 228 188 194 188
query64 2540 732 586 586
query65 4907 4809 4771 4771
query66 1813 480 336 336
query67 29789 29102 29633 29102
query68 3067 1622 962 962
query69 411 291 265 265
query70 1064 978 950 950
query71 297 224 213 213
query72 2916 2535 2268 2268
query73 840 749 426 426
query74 5123 4998 4744 4744
query75 2647 2595 2235 2235
query76 2301 1185 810 810
query77 357 382 316 316
query78 12438 12477 11845 11845
query79 1464 1186 755 755
query80 1081 482 385 385
query81 498 275 241 241
query82 825 153 124 124
query83 350 276 247 247
query84 313 146 113 113
query85 905 519 420 420
query86 418 307 269 269
query87 3404 3319 3186 3186
query88 3668 2784 2770 2770
query89 424 378 331 331
query90 1805 191 187 187
query91 175 156 136 136
query92 65 57 56 56
query93 1495 1452 973 973
query94 625 354 320 320
query95 685 383 355 355
query96 1123 804 377 377
query97 2709 2684 2535 2535
query98 225 226 196 196
query99 1176 1175 1033 1033
Total cold run time: 261445 ms
Total hot run time: 175566 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4725000657
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4743173089
# FE Regression Coverage Report
Increment line coverage ` 90.62% (58/64)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_31f5e7b8e267f8386f1547e9f8cca674c4a7765e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_31f5e7b8e267f8386f1547e9f8cca674c4a7765e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63310:
URL: https://github.com/apache/doris/pull/63310#discussion_r3436948233
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##########
@@ -316,6 +326,7 @@ private void eraseDatabase(long currentTimeMs, int keepNum) {
if (dbInfo == null) {
continue;
}
+ eraseTablesForDatabase(dbInfo);
Database db = dbInfo.getDb();
Review Comment:
This fixes the normal expired-database path, but the parallel same-name database eviction path still bypasses this by-db cleanup. `eraseDatabaseWithSameName()` below still calls `eraseAllTables(dbInfo)`, which only removes tables listed in `RecycleDatabaseInfo`; that list is built from the tables present at database-drop time. Concrete sequence: drop table `t_old` from db `d`, later drop the remaining tables and db `d`, then exceed `max_same_name_catalog_trash_num` with more databases named `d`. The same-name eviction can log `OP_ERASE_DB` for the old db after min latency while `t_old` remains in `idToTable` with a missing db parent. This is distinct from the earlier normal-expired-db thread because that path now calls `eraseTablesForDatabase`, but the same-name db path still uses the old limited helper. Please route `eraseDatabaseWithSameName()` through the same db-id child cleanup and add a same-name database test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4698701401
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4710299170
# FE Regression Coverage Report
Increment line coverage ` 77.78% (7/9)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63310_a15360e85ac725af3a16308e33e1996bd355f920_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63310_a15360e85ac725af3a16308e33e1996bd355f920_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4716448170
<details>
<summary>TPC-H: <b>Total hot run time: 29009 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c777b85bc8a8cb928dba405a4b684eebd819e8b9, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17651 3996 3958 3958
q2 1988 315 191 191
q3 10380 1400 833 833
q4 4679 470 346 346
q5 7482 856 567 567
q6 202 183 145 145
q7 757 847 608 608
q8 10161 1631 1659 1631
q9 6319 4509 4541 4509
q10 6810 1811 1541 1541
q11 438 273 238 238
q12 661 421 296 296
q13 18175 3479 2795 2795
q14 284 268 253 253
q15 q16 786 765 701 701
q17 1609 1191 774 774
q18 6797 5857 5580 5580
q19 2714 1386 1088 1088
q20 514 404 263 263
q21 5964 2572 2403 2403
q22 432 356 289 289
Total cold run time: 104803 ms
Total hot run time: 29009 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4409 4222 4248 4222
q2 330 358 223 223
q3 4624 4973 4399 4399
q4 2032 2164 1371 1371
q5 4422 4293 4286 4286
q6 235 178 125 125
q7 2250 1921 1582 1582
q8 2475 2164 2089 2089
q9 8083 7859 7969 7859
q10 4807 4746 4488 4488
q11 573 410 374 374
q12 760 746 571 571
q13 3242 3743 3008 3008
q14 303 298 298 298
q15 q16 714 715 632 632
q17 1358 1317 1318 1317
q18 7843 7300 6781 6781
q19 1115 1129 1123 1123
q20 2202 2208 1962 1962
q21 5328 4544 4429 4429
q22 520 439 390 390
Total cold run time: 57625 ms
Total hot run time: 51529 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4516384310
<details>
<summary>TPC-H: <b>Total hot run time: 31429 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 98c43256eae708c582b6c51e66fe460dafae051a, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17605 4019 3937 3937
q2 q3 10815 1406 819 819
q4 4690 469 343 343
q5 7567 2258 2107 2107
q6 261 176 135 135
q7 962 764 654 654
q8 9409 1684 1630 1630
q9 6968 4940 4928 4928
q10 6452 2085 1785 1785
q11 452 282 246 246
q12 692 421 293 293
q13 18186 3307 2811 2811
q14 262 253 233 233
q15 q16 815 768 708 708
q17 1002 878 841 841
q18 6835 5801 5583 5583
q19 1183 1237 1229 1229
q20 530 444 281 281
q21 5903 2712 2554 2554
q22 455 384 312 312
Total cold run time: 101044 ms
Total hot run time: 31429 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4579 4627 4723 4627
q2 q3 4946 5180 4618 4618
q4 2128 2189 1409 1409
q5 4756 4740 4703 4703
q6 234 178 134 134
q7 1869 1708 1552 1552
q8 2446 1903 1923 1903
q9 7312 7286 7228 7228
q10 4490 4411 3992 3992
q11 523 383 352 352
q12 704 718 519 519
q13 3049 3334 2865 2865
q14 270 283 248 248
q15 q16 687 715 623 623
q17 1259 1232 1226 1226
q18 7233 6827 6853 6827
q19 1091 1115 1098 1098
q20 2218 2211 1942 1942
q21 5278 4614 4467 4467
q22 537 457 395 395
Total cold run time: 55609 ms
Total hot run time: 50728 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4698517478
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4709445557
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4728290930
<details>
<summary>TPC-H: <b>Total hot run time: 29537 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b977d0b4c19914042cee8e3f06c8358f43aa43a, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17778 4055 4084 4055
q2 2011 318 188 188
q3 10316 1472 823 823
q4 4682 475 341 341
q5 7499 848 568 568
q6 185 170 136 136
q7 770 853 624 624
q8 9363 1630 1638 1630
q9 5719 4524 4551 4524
q10 6759 1790 1540 1540
q11 435 276 243 243
q12 632 420 299 299
q13 18110 3475 2808 2808
q14 272 258 245 245
q15 q16 782 779 706 706
q17 1022 914 959 914
q18 7709 5854 5552 5552
q19 1341 1195 1032 1032
q20 494 427 262 262
q21 6574 2909 2726 2726
q22 458 379 321 321
Total cold run time: 102911 ms
Total hot run time: 29537 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5073 4971 4934 4934
q2 348 382 233 233
q3 4959 5228 4689 4689
q4 2087 2137 1450 1450
q5 4934 4706 4759 4706
q6 236 182 132 132
q7 1855 1749 1545 1545
q8 2406 2096 2127 2096
q9 7580 7403 7361 7361
q10 4758 4632 4222 4222
q11 533 386 356 356
q12 733 743 523 523
q13 3007 3415 2766 2766
q14 278 295 260 260
q15 q16 683 711 632 632
q17 1297 1282 1254 1254
q18 7352 6957 6816 6816
q19 1130 1135 1128 1128
q20 2228 2213 1936 1936
q21 5320 4566 4567 4566
q22 557 447 401 401
Total cold run time: 57354 ms
Total hot run time: 52006 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4732134618
<details>
<summary>TPC-H: <b>Total hot run time: 28900 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c92c60e6d644d9e325f2adc1fcf690eae9e6b7eb, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17664 4075 4042 4042
q2 1987 305 190 190
q3 10372 1381 850 850
q4 4679 469 335 335
q5 7490 869 569 569
q6 193 179 140 140
q7 762 836 617 617
q8 9412 1647 1621 1621
q9 6399 4486 4499 4486
q10 6867 1762 1498 1498
q11 440 274 246 246
q12 633 419 298 298
q13 18141 3369 2838 2838
q14 285 259 244 244
q15 q16 779 781 702 702
q17 1759 1127 763 763
q18 6899 5787 5502 5502
q19 1543 1356 949 949
q20 481 401 271 271
q21 5990 2599 2434 2434
q22 431 358 305 305
Total cold run time: 103206 ms
Total hot run time: 28900 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4393 4296 4269 4269
q2 326 356 233 233
q3 4635 4952 4405 4405
q4 2050 2167 1392 1392
q5 4447 4300 4369 4300
q6 226 175 128 128
q7 1688 2178 1691 1691
q8 2483 2283 2092 2092
q9 7968 7935 7997 7935
q10 4771 4783 4312 4312
q11 574 420 550 420
q12 727 746 546 546
q13 3327 3548 3123 3123
q14 289 304 298 298
q15 q16 732 736 639 639
q17 1386 1396 1362 1362
q18 7964 7552 7017 7017
q19 1153 1090 1099 1090
q20 2222 2212 1939 1939
q21 5292 4607 4553 4553
q22 510 458 410 410
Total cold run time: 57163 ms
Total hot run time: 52154 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4732254714
<details>
<summary>TPC-DS: <b>Total hot run time: 176126 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c92c60e6d644d9e325f2adc1fcf690eae9e6b7eb, data reload: false
query5 4322 637 473 473
query6 428 189 178 178
query7 4827 536 307 307
query8 365 229 196 196
query9 8740 4173 4101 4101
query10 439 301 255 255
query11 5940 2347 2137 2137
query12 158 104 106 104
query13 1348 596 436 436
query14 6403 5418 5112 5112
query14_1 4411 4415 4414 4414
query15 209 200 175 175
query16 996 460 452 452
query17 1133 724 582 582
query18 2576 479 352 352
query19 201 188 149 149
query20 118 109 109 109
query21 220 143 118 118
query22 13791 13623 13490 13490
query23 17515 16623 16239 16239
query23_1 16318 16333 16332 16332
query24 7548 1801 1345 1345
query24_1 1323 1325 1350 1325
query25 565 472 394 394
query26 1314 305 173 173
query27 2681 517 348 348
query28 4464 2055 2082 2055
query29 1088 646 506 506
query30 319 242 197 197
query31 1158 1060 957 957
query32 100 63 62 62
query33 525 325 266 266
query34 1205 1177 688 688
query35 765 786 730 730
query36 1419 1390 1195 1195
query37 160 121 89 89
query38 3280 3155 3059 3059
query39 950 931 883 883
query39_1 878 888 876 876
query40 220 121 102 102
query41 64 62 61 61
query42 98 95 95 95
query43 331 324 289 289
query44 1468 783 771 771
query45 190 184 171 171
query46 1074 1216 760 760
query47 2331 2370 2184 2184
query48 427 421 291 291
query49 617 467 357 357
query50 1023 359 259 259
query51 4301 4275 4252 4252
query52 88 89 77 77
query53 245 268 197 197
query54 270 217 201 201
query55 77 75 74 74
query56 236 233 208 208
query57 1454 1411 1313 1313
query58 244 209 233 209
query59 1575 1616 1468 1468
query60 286 251 228 228
query61 158 146 149 146
query62 697 648 570 570
query63 240 197 198 197
query64 2485 776 616 616
query65 4890 4807 4790 4790
query66 1770 450 377 377
query67 29877 30018 29563 29563
query68 3186 1504 895 895
query69 404 303 272 272
query70 1149 979 992 979
query71 292 236 211 211
query72 2987 2646 2290 2290
query73 877 760 428 428
query74 5120 4973 4803 4803
query75 2622 2603 2212 2212
query76 2284 1206 791 791
query77 376 384 292 292
query78 12522 12537 11876 11876
query79 1480 1150 797 797
query80 1296 476 407 407
query81 530 278 236 236
query82 603 162 125 125
query83 325 277 255 255
query84 268 148 116 116
query85 892 491 402 402
query86 437 293 290 290
query87 3418 3373 3175 3175
query88 3719 2819 2813 2813
query89 437 387 330 330
query90 1877 201 190 190
query91 173 163 138 138
query92 66 58 56 56
query93 1630 1483 859 859
query94 717 347 322 322
query95 673 488 343 343
query96 1075 890 347 347
query97 2675 2683 2560 2560
query98 221 219 202 202
query99 1166 1213 1032 1032
Total cold run time: 262503 ms
Total hot run time: 176126 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4732309940
<details>
<summary>ClickBench: <b>Total hot run time: 25.35 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c92c60e6d644d9e325f2adc1fcf690eae9e6b7eb, data reload: false
query1 0.00 0.00 0.00
query2 0.10 0.05 0.05
query3 0.25 0.14 0.14
query4 1.61 0.15 0.14
query5 0.24 0.23 0.23
query6 1.27 1.07 1.04
query7 0.03 0.01 0.01
query8 0.06 0.04 0.03
query9 0.38 0.32 0.32
query10 0.59 0.56 0.56
query11 0.19 0.14 0.14
query12 0.18 0.14 0.14
query13 0.47 0.47 0.48
query14 1.01 1.01 1.01
query15 0.61 0.60 0.58
query16 0.32 0.32 0.31
query17 1.08 1.10 1.06
query18 0.23 0.21 0.22
query19 2.05 1.96 1.95
query20 0.02 0.01 0.01
query21 15.43 0.19 0.13
query22 4.96 0.06 0.05
query23 16.13 0.31 0.12
query24 2.93 0.42 0.31
query25 0.09 0.06 0.04
query26 0.73 0.22 0.14
query27 0.04 0.04 0.03
query28 3.52 0.89 0.56
query29 12.51 4.34 3.49
query30 0.28 0.14 0.16
query31 2.78 0.61 0.31
query32 3.23 0.60 0.49
query33 3.22 3.29 3.38
query34 15.60 4.18 3.55
query35 3.56 3.51 3.54
query36 0.55 0.43 0.43
query37 0.09 0.07 0.06
query38 0.04 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.16 0.15
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 96.75 s
Total hot run time: 25.35 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4716223332
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4716575397
<details>
<summary>ClickBench: <b>Total hot run time: 25.28 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c777b85bc8a8cb928dba405a4b684eebd819e8b9, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.14 0.14
query4 1.61 0.13 0.13
query5 0.24 0.23 0.23
query6 1.18 1.05 1.04
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.40 0.34 0.31
query10 0.57 0.58 0.56
query11 0.20 0.15 0.14
query12 0.19 0.15 0.15
query13 0.49 0.47 0.47
query14 1.00 1.02 1.03
query15 0.62 0.60 0.60
query16 0.32 0.33 0.33
query17 1.09 1.09 1.13
query18 0.24 0.22 0.22
query19 2.10 2.01 1.91
query20 0.02 0.01 0.00
query21 15.44 0.20 0.13
query22 4.98 0.05 0.05
query23 16.14 0.32 0.12
query24 2.90 0.41 0.33
query25 0.11 0.06 0.04
query26 0.72 0.21 0.14
query27 0.04 0.04 0.04
query28 3.54 0.95 0.52
query29 12.47 4.32 3.49
query30 0.27 0.15 0.15
query31 2.76 0.61 0.32
query32 3.22 0.60 0.49
query33 3.17 3.21 3.26
query34 15.82 4.17 3.50
query35 3.52 3.54 3.51
query36 0.56 0.44 0.43
query37 0.09 0.06 0.07
query38 0.05 0.03 0.04
query39 0.04 0.03 0.03
query40 0.18 0.16 0.16
query41 0.08 0.03 0.03
query42 0.03 0.02 0.02
query43 0.04 0.03 0.04
Total cold run time: 96.9 s
Total hot run time: 25.28 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #63310:
URL: https://github.com/apache/doris/pull/63310#issuecomment-4725167466
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman closed pull request #63310: [fix](fe) Fix stale timestamp in CatalogRecycleBin erase daemon
URL: https://github.com/apache/doris/pull/63310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org