You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2026/10/22 07:19:38 UTC
[PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
airborne12 opened a new pull request, #69441:
URL: https://github.com/apache/doris/pull/69441
### What problem does this PR solve?
Issue Number: None
Related PR: #68674
Problem Summary:
The light `DROP INDEX` pre-check added in #68674 (`SchemaChangeHandler#checkNoIndexChangeJobOnPartitions`) calls `hasIndexChangeJobOnPartition` once per partition, and every call copies and scans the whole retained `indexChangeJobs` history (finished jobs are kept for days). For a table with P partitions and a history of H jobs that is O(P * H) work while the table write lock is held, on top of the existing per-partition check in `buildOrDeleteTableInvertedIndices`. A recent `BUILD INDEX` on a table with many partitions leaves at least one record per partition, which makes it quadratic in the number of partitions.
The pre-check also reported a conflict with the `BUILD INDEX` wording `partition xxx has been built specified index. please check your build stmt.`, which is misleading for a `DROP INDEX`.
Fix: the pre-check takes one snapshot of the job history and scans it once, matching jobs on any of the table's partitions. The match condition is extracted into `isConflictingIndexChangeJob` and shared with `hasIndexChangeJobOnPartition`, so both checks keep the same rule. The pre-check reports `partition xxx of table t has an unfinished index change job, please retry after it finishes.` The `BUILD INDEX` / `DROP INDEX ... PARTITION` check and its message are unchanged.
### Release note
A light DROP INDEX rejected because of an unfinished index change job now reports `partition xxx of table t has an unfinished index change job, please retry after it finishes.`
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
`SchemaChangeHandlerTest#testDropIndexRejectedByIndexChangeJobKeepsMetadata` now expects the new message, and keeps checking the rejected DROP leaves the metadata unchanged and the drop succeeds once the job finishes.
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
Only the error message of a rejected light DROP INDEX changes.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6354976212
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69441:
URL: https://github.com/apache/doris/pull/69441#discussion_r4345804049
##########
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##########
@@ -3974,20 +3976,25 @@ public boolean hasIndexChangeJobOnPartition(
List<Index> alterIndexes, boolean isDrop) {
// TODO: this is temporary methods
for (IndexChangeJob indexChangeJob : ImmutableList.copyOf(indexChangeJobs.values())) {
- if (indexChangeJob.getOriginIndexId() == originIndexId
- && indexChangeJob.getDbId() == dbId
- && indexChangeJob.getTableId() == tableId
- && indexChangeJob.getPartitionName().equals(partitionName)
- && ((Config.isNotCloudMode() && indexChangeJob.hasSameAlterInvertedIndex(isDrop, alterIndexes))
- || Config.isCloudMode())
- && !indexChangeJob.isDone()) {
- // if JobState is done (CANCELLED or FINISHED), also allow user to create job again
+ if (isConflictingIndexChangeJob(indexChangeJob, originIndexId, dbId, tableId, alterIndexes, isDrop)
Review Comment:
[P2] Keep the partition guard ahead of index-list matching
`hasIndexChangeJobOnPartition` is still called once per selected partition by `buildOrDeleteTableInvertedIndices`. Moving the partition comparison after `isConflictingIndexChangeJob` makes local mode run `hasSameAlterInvertedIndex`'s nested index scans for every retained job on the table, including finished jobs, for every partition. For example, after a batch partition DROP of 50 indexes across 1,000 partitions completes, dropping a disjoint batch of 50 indexes across those partitions now performs about 2.5 billion index-ID comparisons against that history instead of 2.5 million (in addition to the existing history scans). This work holds the table write lock and can substantially stall metadata access. Preserve the partition-name short circuit before invoking the shared conflict predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6344852067
# FE Regression Coverage Report
Increment line coverage ` 7.53% (7/93)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69441_89555012934acca4763f0176649e72828b41496d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69441_89555012934acca4763f0176649e72828b41496d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6350036463
# FE Regression Coverage Report
Increment line coverage ` 50.00% (7/14)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69441_89555012934acca4763f0176649e72828b41496d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69441_89555012934acca4763f0176649e72828b41496d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6340169780
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6335117199
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6349898369
# FE UT Coverage Report
Increment line coverage ` 57.14% (8/14)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69441_89555012934acca4763f0176649e72828b41496d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69441_89555012934acca4763f0176649e72828b41496d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6350315155
# FE Regression Coverage Report
Increment line coverage ` 50.00% (7/14)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69441_89555012934acca4763f0176649e72828b41496d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69441_89555012934acca4763f0176649e72828b41496d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 merged PR #69441:
URL: https://github.com/apache/doris/pull/69441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6348470285
<details>
<summary>ClickBench: <b>Total hot run time: 24.4 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 89555012934acca4763f0176649e72828b41496d, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.09 0.09
query4 1.61 0.10 0.10
query5 0.24 0.25 0.25
query6 1.17 0.70 0.68
query7 0.04 0.01 0.01
query8 0.06 0.05 0.04
query9 0.41 0.36 0.34
query10 0.59 0.61 0.59
query11 0.20 0.15 0.15
query12 0.18 0.16 0.15
query13 0.52 0.50 0.51
query14 1.03 1.01 1.02
query15 0.64 0.63 0.63
query16 0.35 0.36 0.34
query17 1.10 1.16 1.15
query18 0.22 0.20 0.20
query19 2.09 1.93 2.01
query20 0.02 0.01 0.02
query21 15.52 0.19 0.13
query22 4.97 0.06 0.05
query23 16.25 0.32 0.12
query24 3.09 0.61 0.36
query25 0.11 0.05 0.06
query26 1.16 0.22 0.16
query27 0.06 0.06 0.05
query28 3.15 0.77 0.38
query29 12.51 4.12 3.26
query30 0.32 0.15 0.12
query31 2.78 0.57 0.32
query32 3.23 0.62 0.50
query33 3.16 3.21 3.17
query34 15.49 4.03 3.37
query35 3.34 3.32 3.37
query36 0.57 0.46 0.44
query37 0.09 0.07 0.06
query38 0.04 0.04 0.04
query39 0.04 0.03 0.03
query40 0.18 0.15 0.15
query41 0.09 0.03 0.04
query42 0.04 0.03 0.02
query43 0.05 0.04 0.04
Total cold run time: 97.08 s
Total hot run time: 24.4 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on code in PR #69441:
URL: https://github.com/apache/doris/pull/69441#discussion_r4347840078
##########
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##########
@@ -3974,20 +3976,25 @@ public boolean hasIndexChangeJobOnPartition(
List<Index> alterIndexes, boolean isDrop) {
// TODO: this is temporary methods
for (IndexChangeJob indexChangeJob : ImmutableList.copyOf(indexChangeJobs.values())) {
- if (indexChangeJob.getOriginIndexId() == originIndexId
- && indexChangeJob.getDbId() == dbId
- && indexChangeJob.getTableId() == tableId
- && indexChangeJob.getPartitionName().equals(partitionName)
- && ((Config.isNotCloudMode() && indexChangeJob.hasSameAlterInvertedIndex(isDrop, alterIndexes))
- || Config.isCloudMode())
- && !indexChangeJob.isDone()) {
- // if JobState is done (CANCELLED or FINISHED), also allow user to create job again
+ if (isConflictingIndexChangeJob(indexChangeJob, originIndexId, dbId, tableId, alterIndexes, isDrop)
Review Comment:
Fixed in the latest commit: both the per-partition check (hasIndexChangeJobOnPartition) and the single-scan pre-check compare the partition first again and only then run the shared predicate with the index-list comparison.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6341649231
<details>
<summary>ClickBench: <b>Total hot run time: 24.29 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 89555012934acca4763f0176649e72828b41496d, data reload: false
query1 0.00 0.00 0.00
query2 0.09 0.05 0.05
query3 0.26 0.09 0.08
query4 1.60 0.13 0.11
query5 0.23 0.23 0.24
query6 1.17 0.69 0.70
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.41 0.36 0.34
query10 0.56 0.56 0.60
query11 0.19 0.14 0.14
query12 0.18 0.14 0.14
query13 0.48 0.50 0.49
query14 1.03 1.02 1.02
query15 0.63 0.61 0.60
query16 0.33 0.34 0.34
query17 1.17 1.12 1.10
query18 0.21 0.19 0.19
query19 2.03 1.96 1.91
query20 0.02 0.01 0.02
query21 15.49 0.19 0.14
query22 4.96 0.05 0.05
query23 16.27 0.32 0.12
query24 3.04 0.54 0.51
query25 0.11 0.06 0.06
query26 1.16 0.21 0.14
query27 0.04 0.04 0.05
query28 3.73 0.78 0.35
query29 12.48 4.08 3.26
query30 0.28 0.13 0.09
query31 2.76 0.55 0.32
query32 3.23 0.59 0.50
query33 3.18 3.18 3.17
query34 15.33 3.99 3.37
query35 3.35 3.35 3.34
query36 0.55 0.48 0.42
query37 0.09 0.07 0.06
query38 0.04 0.04 0.04
query39 0.03 0.03 0.04
query40 0.18 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 97.15 s
Total hot run time: 24.29 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6348362325
<details>
<summary>TPC-DS: <b>Total hot run time: 151565 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 89555012934acca4763f0176649e72828b41496d, data reload: false
query5 4313 586 460 460
query6 441 208 194 194
query7 4806 451 224 224
query8 327 179 171 171
query9 8797 3110 3133 3110
query10 455 318 261 261
query11 5917 3912 3597 3597
query12 145 95 86 86
query13 1245 441 319 319
query14 6135 3938 3718 3718
query14_1 3518 3527 3527 3527
query15 205 196 180 180
query16 945 423 392 392
query17 832 640 510 510
query18 2407 417 303 303
query19 197 158 135 135
query20 84 81 78 78
query21 207 132 108 108
query22 12964 13039 12911 12911
query23 13406 12813 12510 12510
query23_1 12505 12524 12584 12524
query24 7383 851 484 484
query24_1 469 475 469 469
query25 516 401 343 343
query26 1260 260 140 140
query27 2770 437 270 270
query28 4625 1951 1951 1951
query29 1567 607 460 460
query30 295 222 184 184
query31 1018 926 819 819
query32 149 89 96 89
query33 553 302 247 247
query34 932 871 506 506
query35 739 767 658 658
query36 805 803 707 707
query37 130 105 87 87
query38 1798 1747 1679 1679
query39 792 785 747 747
query39_1 731 740 756 740
query40 220 119 101 101
query41 75 73 67 67
query42 90 87 87 87
query43 355 367 324 324
query44 1378 686 693 686
query45 183 178 174 174
query46 842 981 588 588
query47 2922 2747 2660 2660
query48 300 330 233 233
query49 587 409 293 293
query50 678 282 219 219
query51 10357 10497 10327 10327
query52 81 88 74 74
query53 193 222 162 162
query54 269 212 189 189
query55 75 70 63 63
query56 267 246 219 219
query57 1655 1636 1565 1565
query58 290 279 275 275
query59 2132 2164 1954 1954
query60 307 260 238 238
query61 174 166 167 166
query62 405 339 287 287
query63 196 168 169 168
query64 2904 1076 768 768
query65 3315 3287 3271 3271
query66 1779 431 297 297
query67 20059 20153 20026 20026
query68 3017 1027 608 608
query69 385 295 258 258
query70 867 788 752 752
query71 302 235 215 215
query72 2706 2640 2398 2398
query73 529 543 304 304
query74 4644 4510 4377 4377
query75 2355 2301 1921 1921
query76 2320 1017 607 607
query77 345 377 275 275
query78 9744 9764 9174 9174
query79 1027 852 519 519
query80 1239 456 357 357
query81 599 322 271 271
query82 599 130 110 110
query83 298 200 179 179
query84 305 120 107 107
query85 859 437 385 385
query86 397 243 232 232
query87 2031 1968 1812 1812
query88 3771 2685 2672 2672
query89 328 286 248 248
query90 1860 177 172 172
query91 169 156 130 130
query92 102 89 83 83
query93 1138 1030 565 565
query94 628 324 277 277
query95 571 338 359 338
query96 675 507 235 235
query97 2458 2454 2331 2331
query98 161 171 149 149
query99 761 758 638 638
Total cold run time: 235436 ms
Total hot run time: 151565 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6348158168
<details>
<summary>TPC-H: <b>Total hot run time: 29029 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 89555012934acca4763f0176649e72828b41496d, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17602 5115 5097 5097
q2 2145 346 279 279
q3 10147 1391 843 843
q4 4681 463 347 347
q5 7448 811 541 541
q6 184 188 159 159
q7 753 812 606 606
q8 9254 1360 1090 1090
q9 5549 4199 4199 4199
q10 6829 1293 1022 1022
q11 430 253 240 240
q12 639 408 303 303
q13 18079 3702 3029 3029
q14 276 273 255 255
q15 q16 731 714 670 670
q17 1198 762 583 583
q18 7106 6176 6114 6114
q19 1247 977 601 601
q20 393 352 225 225
q21 5918 2536 2533 2533
q22 388 340 293 293
Total cold run time: 100997 ms
Total hot run time: 29029 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6309 5914 6113 5914
q2 710 604 595 595
q3 5477 5759 5242 5242
q4 2041 2192 1649 1649
q5 5801 5686 5872 5686
q6 247 184 142 142
q7 2215 2008 1806 1806
q8 3076 2680 2662 2662
q9 7881 7951 7681 7681
q10 3642 3596 3158 3158
q11 603 449 407 407
q12 678 686 513 513
q13 3365 3682 3073 3073
q14 290 289 268 268
q15 q16 654 700 633 633
q17 7801 7050 6989 6989
q18 12968 12182 12939 12182
q19 934 840 820 820
q20 2153 2157 1946 1946
q21 5938 4935 4990 4935
q22 492 487 404 404
Total cold run time: 73275 ms
Total hot run time: 66705 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6341349652
<details>
<summary>TPC-H: <b>Total hot run time: 28191 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 89555012934acca4763f0176649e72828b41496d, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17572 5101 5016 5016
q2 2123 298 247 247
q3 10276 1340 818 818
q4 4674 461 333 333
q5 7416 791 523 523
q6 180 186 153 153
q7 754 783 608 608
q8 9253 1336 1028 1028
q9 5417 4149 4162 4149
q10 6810 1293 1009 1009
q11 425 249 234 234
q12 652 394 279 279
q13 18098 3522 2978 2978
q14 278 276 245 245
q15 q16 724 698 650 650
q17 1176 737 568 568
q18 6950 6157 6088 6088
q19 1109 946 622 622
q20 370 338 231 231
q21 5166 2147 2292 2147
q22 343 301 265 265
Total cold run time: 99766 ms
Total hot run time: 28191 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5551 5574 5571 5571
q2 688 585 549 549
q3 5185 5377 4933 4933
q4 1985 2135 1547 1547
q5 5250 5137 5442 5137
q6 262 214 156 156
q7 2383 2008 1786 1786
q8 2960 2662 2696 2662
q9 7845 8007 7860 7860
q10 3682 3656 3222 3222
q11 599 426 408 408
q12 789 754 525 525
q13 3585 3791 3239 3239
q14 326 345 302 302
q15 q16 717 734 648 648
q17 8268 7333 6835 6835
q18 13022 12225 12845 12225
q19 876 758 782 758
q20 2151 2131 1922 1922
q21 5587 5142 4648 4648
q22 484 447 398 398
Total cold run time: 72195 ms
Total hot run time: 65331 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6341550709
<details>
<summary>TPC-DS: <b>Total hot run time: 150195 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 89555012934acca4763f0176649e72828b41496d, data reload: false
query5 4315 590 443 443
query6 419 212 190 190
query7 4809 436 228 228
query8 332 162 150 150
query9 8759 3061 3086 3061
query10 474 303 252 252
query11 5998 3842 3596 3596
query12 149 92 90 90
query13 1247 450 318 318
query14 6053 3961 3642 3642
query14_1 3421 3407 3423 3407
query15 216 194 178 178
query16 940 428 412 412
query17 848 647 538 538
query18 2424 444 312 312
query19 197 167 158 158
query20 83 85 83 83
query21 213 133 107 107
query22 12945 12892 12908 12892
query23 13426 12867 12406 12406
query23_1 12362 12547 12468 12468
query24 7397 823 457 457
query24_1 465 470 479 470
query25 531 420 359 359
query26 1256 256 142 142
query27 2765 435 266 266
query28 4635 1934 1935 1934
query29 1587 623 458 458
query30 295 221 188 188
query31 997 927 812 812
query32 145 98 93 93
query33 532 302 242 242
query34 881 849 497 497
query35 725 756 668 668
query36 805 787 710 710
query37 124 100 83 83
query38 1768 1729 1709 1709
query39 791 809 745 745
query39_1 735 727 738 727
query40 219 117 96 96
query41 72 66 66 66
query42 90 88 83 83
query43 343 382 320 320
query44 1319 694 689 689
query45 178 174 164 164
query46 830 984 567 567
query47 2829 2838 2641 2641
query48 306 296 228 228
query49 586 393 292 292
query50 689 282 208 208
query51 10561 10502 10469 10469
query52 84 78 75 75
query53 182 207 150 150
query54 245 203 174 174
query55 70 67 64 64
query56 226 206 197 197
query57 1676 1598 1570 1570
query58 285 262 258 258
query59 2136 2171 1910 1910
query60 285 234 215 215
query61 144 139 141 139
query62 378 330 281 281
query63 184 155 157 155
query64 2733 912 737 737
query65 3311 3253 3255 3253
query66 1786 417 293 293
query67 20027 20365 19780 19780
query68 2869 1039 618 618
query69 385 297 240 240
query70 857 781 756 756
query71 291 231 215 215
query72 2672 2334 2273 2273
query73 507 519 295 295
query74 4617 4562 4343 4343
query75 2313 2279 1918 1918
query76 2296 1017 602 602
query77 332 362 266 266
query78 9596 9766 9120 9120
query79 977 851 518 518
query80 816 409 342 342
query81 574 314 273 273
query82 571 134 105 105
query83 283 191 182 182
query84 301 118 107 107
query85 938 417 370 370
query86 354 232 227 227
query87 1989 1972 1790 1790
query88 3626 2681 2651 2651
query89 321 277 237 237
query90 1718 168 161 161
query91 160 151 129 129
query92 93 89 91 89
query93 985 989 568 568
query94 497 305 265 265
query95 562 329 296 296
query96 651 471 229 229
query97 2429 2448 2294 2294
query98 161 150 148 148
query99 774 758 639 639
Total cold run time: 233085 ms
Total hot run time: 150195 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6351215082
run check_coverage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6335118323
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6335213708
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69441:
URL: https://github.com/apache/doris/pull/69441#issuecomment-6342759013
# FE UT Coverage Report
Increment line coverage ` 57.14% (8/14)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69441_89555012934acca4763f0176649e72828b41496d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69441_89555012934acca4763f0176649e72828b41496d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org