You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2026/10/22 07:20:27 UTC
[PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
airborne12 opened a new pull request, #69444:
URL: https://github.com/apache/doris/pull/69444
### What problem does this PR solve?
Issue Number: None
Related PR: #68674
Problem Summary:
When an ALTER TABLE statement combines an index change that can run on the light index change path with a clause that needs a heavy schema change, the light path wins and the heavy work is never done:
```sql
-- v was INT, idx is an INVERTED index: the shadow column __doris_shadow_v BIGINT is published and the
-- schema change job is recorded as FINISHED, but no BE conversion job runs
ALTER TABLE t MODIFY COLUMN v BIGINT NULL, DROP INDEX idx;
ALTER TABLE t MODIFY COLUMN v BIGINT NULL, ADD INDEX i2 (x) USING INVERTED;
-- enable_add_index_for_new_data = false: bf_v2 must be built for the existing data, but it is only
-- published for new data
ALTER TABLE t ADD INDEX bf_v2 (v2) USING BLOOMFILTER, DROP INDEX idx;
```
The same holds for every column or property clause that cannot run as a light schema change (dropping a key column, reordering columns, sequence mapping tables, table properties). On a changelog table such a statement also bypassed the check that only light changes are allowed, because that check looked at the candidate flag `lightIndexChange`.
Root cause: in `SchemaChangeHandler#process` every clause that is not light sets `lightSchemaChange = false`, index clauses included, so nothing records that a clause needs a heavy schema change. The dispatch then takes the light index change branch whenever an index clause can be light.
Fix: the clause loop also records `requiresHeavyChange` for column and property clauses that cannot be light and for an `ADD INDEX` that cannot be light. After the loop, the light index change path is only used when no clause needs heavy work; otherwise the statement runs as a heavy schema change job (`createJob`), which applies the index changes as well, exactly as it does when `enable_light_index_change` is off. The changelog check uses the path that is actually taken. No statement is newly rejected (other than on changelog tables, where the heavy path was never allowed).
Notes:
- A `DROP INDEX` without light support still needs no heavy work: in a light index change only its metadata is dropped (same-statement NGRAM_BF to BLOOMFILTER replacement in local mode keeps working).
- The light DROP INDEX pre-check from #68674 still runs before the clause loop, because whether a clause needs heavy work is only known after it is processed. In cloud mode it therefore still requires a compute group for such a statement, as before this change.
### Release note
ALTER TABLE statements that combine a light index change with a column, property or index change that needs a heavy schema change now run as a heavy schema change job, instead of publishing the new schema without converting the data or building the index.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
`SchemaChangeHandlerTest#testLightIndexChangeWithHeavyWorkRunsHeavySchemaChange` runs `MODIFY COLUMN vi BIGINT NULL, DROP INDEX idx_v1` and, with `enable_add_index_for_new_data = false`, `ADD INDEX bf_v2 (v2) USING BLOOMFILTER, DROP INDEX idx_v3`. It checks each creates one schema change job and no index change job, and after the jobs finish that `vi` is BIGINT, no shadow column is left and the index list is as expected. The test fails without the change.
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
Such statements run as a heavy schema change job instead of the light index change path.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6335132025
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 merged PR #69444:
URL: https://github.com/apache/doris/pull/69444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6341615580
<details>
<summary>TPC-H: <b>Total hot run time: 28790 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e7c0c1f0c37fd54cde6472b734098d918d224883, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17585 5243 5101 5101
q2 2124 323 267 267
q3 10199 1443 824 824
q4 4678 462 338 338
q5 7427 805 520 520
q6 178 182 152 152
q7 748 825 585 585
q8 9253 1358 1094 1094
q9 5749 4210 4201 4201
q10 6829 1296 1008 1008
q11 456 245 227 227
q12 640 399 280 280
q13 18080 3599 3053 3053
q14 273 271 248 248
q15 q16 728 713 646 646
q17 1222 750 590 590
q18 6941 6230 6149 6149
q19 1249 947 597 597
q20 374 333 227 227
q21 5853 2417 2469 2417
q22 362 314 266 266
Total cold run time: 100948 ms
Total hot run time: 28790 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5968 5983 6066 5983
q2 724 566 635 566
q3 5584 5786 5324 5324
q4 1989 2146 1590 1590
q5 5648 5769 5476 5476
q6 241 187 142 142
q7 2217 2006 1857 1857
q8 3040 2703 2671 2671
q9 7981 7862 7540 7540
q10 3635 3579 3121 3121
q11 596 413 377 377
q12 660 656 502 502
q13 3294 3589 3047 3047
q14 288 293 264 264
q15 q16 662 689 608 608
q17 7743 7054 6901 6901
q18 12959 12216 12867 12216
q19 883 771 784 771
q20 2153 2139 1914 1914
q21 5731 4831 4857 4831
q22 505 462 430 430
Total cold run time: 72501 ms
Total hot run time: 66131 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6344721507
# FE Regression Coverage Report
Increment line coverage ` 12.50% (11/88)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69444_e7c0c1f0c37fd54cde6472b734098d918d224883_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69444_e7c0c1f0c37fd54cde6472b734098d918d224883_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69444:
URL: https://github.com/apache/doris/pull/69444#discussion_r4345828438
##########
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##########
@@ -2664,7 +2679,12 @@ public int getAsInt() {
lightIndexChange, buildIndexChange, indexSchemaMap);
}
- if (olapTable.hasChangelog() && !(lightSchemaChange || lightIndexChange)) {
+ // The light index change path only publishes the new schema and submits index delete jobs. When a
+ // clause also needs a heavy schema change, the whole statement runs as a heavy schema change job,
+ // which applies the index changes as well.
+ boolean useLightIndexChange = Config.enable_light_index_change && lightIndexChange
+ && !requiresHeavyChange;
+ if (olapTable.hasChangelog() && !(lightSchemaChange || useLightIndexChange)) {
Review Comment:
[P2] Preserve bloom-filter metadata when rejecting a mixed ALTER
On a local changelog DUPLICATE KEY table with `bloom_filter_columns='v1'`, an inverted `idx_v3(v3)`, and `enable_add_index_for_new_data=false`, `ALTER TABLE t DROP COLUMN v1, ADD INDEX bf_v2(v2) USING BLOOMFILTER, DROP INDEX idx_v3` now hits this exception. However, `processDropColumn()` has already called `olapTable.setBloomFilterInfo(...)` on the live table, and the exception only unlocks it. The rejected ALTER therefore leaves `v1` and `idx_v3` present but silently removes `v1` from the bloom-filter configuration without a job or journal entry, so followers retain different metadata. Stage these metadata updates until validation succeeds, or restore them when the statement is rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6432585250
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6335130317
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6418413157
<details>
<summary>ClickBench: <b>Total hot run time: 24.22 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 190160289cb0eec891e5b5ee3f88f8de0a655896, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.04
query3 0.26 0.09 0.09
query4 1.60 0.11 0.12
query5 0.26 0.24 0.24
query6 1.19 0.70 0.69
query7 0.04 0.00 0.00
query8 0.05 0.04 0.03
query9 0.40 0.33 0.36
query10 0.57 0.59 0.58
query11 0.19 0.14 0.14
query12 0.18 0.15 0.14
query13 0.49 0.49 0.48
query14 1.02 1.00 0.99
query15 0.64 0.61 0.60
query16 0.34 0.34 0.35
query17 1.16 1.19 1.13
query18 0.21 0.19 0.18
query19 2.11 2.09 2.01
query20 0.03 0.02 0.01
query21 15.52 0.19 0.14
query22 4.85 0.06 0.06
query23 16.26 0.34 0.12
query24 3.06 0.48 0.36
query25 0.09 0.07 0.06
query26 0.83 0.22 0.15
query27 0.06 0.03 0.04
query28 3.33 0.76 0.33
query29 12.49 4.15 3.26
query30 0.28 0.14 0.09
query31 2.78 0.56 0.32
query32 3.22 0.60 0.50
query33 3.17 3.26 3.15
query34 15.42 4.00 3.39
query35 3.33 3.33 3.37
query36 0.57 0.42 0.44
query37 0.10 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 96.63 s
Total hot run time: 24.22 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6335195721
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6415879821
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6341815277
<details>
<summary>TPC-DS: <b>Total hot run time: 150677 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e7c0c1f0c37fd54cde6472b734098d918d224883, data reload: false
query5 4317 587 425 425
query6 425 202 193 193
query7 4802 444 225 225
query8 321 179 148 148
query9 8787 3084 3122 3084
query10 467 298 297 297
query11 6045 4043 3569 3569
query12 142 90 87 87
query13 1256 437 325 325
query14 6095 3920 3621 3621
query14_1 3409 3398 3400 3398
query15 205 196 180 180
query16 946 423 424 423
query17 858 665 543 543
query18 2423 432 305 305
query19 205 163 137 137
query20 85 83 84 83
query21 211 145 106 106
query22 12910 13092 12748 12748
query23 13592 12786 12571 12571
query23_1 12642 12470 12645 12470
query24 7382 819 465 465
query24_1 484 479 477 477
query25 534 416 364 364
query26 1257 254 143 143
query27 2775 433 271 271
query28 4630 1942 1944 1942
query29 1585 608 453 453
query30 301 221 188 188
query31 1000 925 818 818
query32 154 94 93 93
query33 543 309 238 238
query34 893 844 497 497
query35 741 768 657 657
query36 808 804 747 747
query37 130 100 82 82
query38 1764 1727 1671 1671
query39 793 787 747 747
query39_1 735 734 738 734
query40 223 118 95 95
query41 71 72 72 72
query42 102 85 83 83
query43 345 375 313 313
query44 1305 696 684 684
query45 177 191 175 175
query46 832 966 566 566
query47 2931 2961 2658 2658
query48 283 301 212 212
query49 568 386 285 285
query50 684 273 204 204
query51 10273 10479 10713 10479
query52 81 79 69 69
query53 179 209 156 156
query54 236 194 174 174
query55 71 74 61 61
query56 226 210 218 210
query57 1621 1616 1558 1558
query58 279 256 263 256
query59 2168 2138 1988 1988
query60 277 238 228 228
query61 148 145 142 142
query62 378 350 276 276
query63 187 160 155 155
query64 2749 959 767 767
query65 3298 3248 3257 3248
query66 1817 426 302 302
query67 20168 20250 19960 19960
query68 2945 989 605 605
query69 387 287 250 250
query70 858 774 769 769
query71 297 245 224 224
query72 2733 2550 2297 2297
query73 507 516 290 290
query74 4607 4529 4362 4362
query75 2312 2296 1902 1902
query76 2324 999 595 595
query77 340 384 276 276
query78 9796 9675 9139 9139
query79 1263 833 520 520
query80 1237 420 346 346
query81 596 315 279 279
query82 568 133 105 105
query83 304 194 181 181
query84 315 123 109 109
query85 866 425 382 382
query86 395 237 235 235
query87 1988 1922 1790 1790
query88 3695 2646 2622 2622
query89 330 280 249 249
query90 1918 172 165 165
query91 166 150 129 129
query92 98 90 91 90
query93 1136 989 569 569
query94 645 311 275 275
query95 575 373 298 298
query96 667 492 233 233
query97 2454 2435 2342 2342
query98 161 155 143 143
query99 765 764 649 649
Total cold run time: 235415 ms
Total hot run time: 150677 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6415881676
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6418063383
<details>
<summary>TPC-H: <b>Total hot run time: 28202 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 190160289cb0eec891e5b5ee3f88f8de0a655896, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17578 5130 5048 5048
q2 2202 303 244 244
q3 10196 1359 814 814
q4 4682 460 336 336
q5 7429 795 524 524
q6 183 187 152 152
q7 755 782 600 600
q8 9257 1331 1033 1033
q9 5478 4144 4132 4132
q10 6807 1292 1019 1019
q11 427 248 232 232
q12 638 393 273 273
q13 18105 3555 3021 3021
q14 268 276 253 253
q15 q16 727 708 657 657
q17 1180 743 571 571
q18 6986 6153 6136 6136
q19 1117 964 592 592
q20 383 330 221 221
q21 4862 2095 2314 2095
q22 338 292 249 249
Total cold run time: 99598 ms
Total hot run time: 28202 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5580 5610 5605 5605
q2 674 565 540 540
q3 5195 5384 4910 4910
q4 2024 2124 1527 1527
q5 5281 5174 5490 5174
q6 242 199 152 152
q7 2390 2049 1885 1885
q8 2990 2693 2714 2693
q9 8151 7994 7995 7994
q10 3688 3766 3244 3244
q11 612 449 424 424
q12 721 701 535 535
q13 3493 4051 3296 3296
q14 314 316 272 272
q15 q16 721 724 644 644
q17 8418 7063 6933 6933
q18 13078 12267 12806 12267
q19 900 768 768 768
q20 2157 2152 1907 1907
q21 5606 4581 4690 4581
q22 489 439 405 405
Total cold run time: 72724 ms
Total hot run time: 65756 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6432584614
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6341916848
<details>
<summary>ClickBench: <b>Total hot run time: 24.38 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e7c0c1f0c37fd54cde6472b734098d918d224883, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.10 0.09
query4 1.61 0.10 0.11
query5 0.24 0.24 0.24
query6 1.18 0.69 0.67
query7 0.04 0.01 0.01
query8 0.06 0.04 0.03
query9 0.40 0.35 0.36
query10 0.58 0.55 0.59
query11 0.20 0.15 0.16
query12 0.19 0.15 0.16
query13 0.50 0.49 0.49
query14 1.01 1.02 1.00
query15 0.63 0.60 0.61
query16 0.36 0.35 0.35
query17 1.13 1.09 1.10
query18 0.21 0.19 0.19
query19 2.05 1.98 2.02
query20 0.02 0.01 0.01
query21 15.42 0.19 0.14
query22 4.93 0.05 0.06
query23 16.30 0.33 0.12
query24 2.74 0.74 0.42
query25 0.09 0.06 0.05
query26 1.47 0.23 0.17
query27 0.04 0.04 0.04
query28 3.64 0.83 0.38
query29 12.52 4.10 3.25
query30 0.28 0.13 0.10
query31 2.77 0.57 0.33
query32 3.24 0.59 0.49
query33 3.14 3.27 3.20
query34 15.31 4.00 3.40
query35 3.32 3.35 3.33
query36 0.54 0.44 0.43
query37 0.10 0.06 0.06
query38 0.06 0.04 0.04
query39 0.04 0.03 0.03
query40 0.19 0.15 0.15
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.03 0.04
Total cold run time: 97.1 s
Total hot run time: 24.38 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6418292407
<details>
<summary>TPC-DS: <b>Total hot run time: 150575 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 190160289cb0eec891e5b5ee3f88f8de0a655896, data reload: false
query5 4303 585 436 436
query6 425 212 189 189
query7 4785 466 226 226
query8 321 156 155 155
query9 8755 3089 3103 3089
query10 472 304 256 256
query11 5932 3864 3561 3561
query12 142 94 87 87
query13 1275 465 328 328
query14 6074 3896 3646 3646
query14_1 3418 3432 3405 3405
query15 211 200 178 178
query16 947 425 422 422
query17 859 663 532 532
query18 2406 414 317 317
query19 221 162 137 137
query20 86 85 84 84
query21 214 138 114 114
query22 13039 12932 12683 12683
query23 13411 12946 12428 12428
query23_1 12549 12688 12593 12593
query24 7397 813 457 457
query24_1 471 478 473 473
query25 533 425 361 361
query26 1257 256 144 144
query27 2759 438 273 273
query28 4609 1914 1916 1914
query29 1568 623 444 444
query30 300 223 186 186
query31 1016 924 820 820
query32 179 94 95 94
query33 542 306 236 236
query34 910 850 514 514
query35 752 766 654 654
query36 868 828 725 725
query37 123 102 80 80
query38 1822 1846 1696 1696
query39 788 806 746 746
query39_1 733 740 748 740
query40 221 117 99 99
query41 76 70 68 68
query42 90 88 87 87
query43 348 376 314 314
query44 1339 718 718 718
query45 189 188 169 169
query46 842 984 575 575
query47 2778 2890 2373 2373
query48 286 301 213 213
query49 589 401 292 292
query50 665 268 214 214
query51 10629 10612 10579 10579
query52 76 82 68 68
query53 190 205 153 153
query54 235 191 176 176
query55 68 68 62 62
query56 227 226 207 207
query57 1645 1613 1543 1543
query58 288 259 257 257
query59 2194 2265 1975 1975
query60 289 238 230 230
query61 149 142 145 142
query62 382 334 305 305
query63 189 160 159 159
query64 2725 951 756 756
query65 3299 3261 3260 3260
query66 1779 428 296 296
query67 20263 20310 20077 20077
query68 3040 1000 618 618
query69 381 280 249 249
query70 841 775 780 775
query71 306 237 211 211
query72 2661 2585 2258 2258
query73 509 534 286 286
query74 4612 4531 4336 4336
query75 2323 2302 1896 1896
query76 2333 1010 594 594
query77 353 368 263 263
query78 9825 9782 9183 9183
query79 994 846 537 537
query80 1257 417 345 345
query81 586 313 271 271
query82 587 132 113 113
query83 282 191 196 191
query84 336 127 102 102
query85 1086 415 371 371
query86 362 237 227 227
query87 1966 1958 1813 1813
query88 3647 2659 2652 2652
query89 323 283 243 243
query90 1711 170 167 167
query91 168 149 126 126
query92 100 82 87 82
query93 992 987 576 576
query94 537 323 269 269
query95 565 324 359 324
query96 655 489 229 229
query97 2444 2439 2338 2338
query98 159 152 146 146
query99 774 763 647 647
Total cold run time: 234997 ms
Total hot run time: 150575 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69444:
URL: https://github.com/apache/doris/pull/69444#issuecomment-6422871218
# FE Regression Coverage Report
Increment line coverage ` 55.00% (11/20)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69444_190160289cb0eec891e5b5ee3f88f8de0a655896_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69444_190160289cb0eec891e5b5ee3f88f8de0a655896_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on code in PR #69444:
URL: https://github.com/apache/doris/pull/69444#discussion_r4347852512
##########
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##########
@@ -2664,7 +2679,12 @@ public int getAsInt() {
lightIndexChange, buildIndexChange, indexSchemaMap);
}
- if (olapTable.hasChangelog() && !(lightSchemaChange || lightIndexChange)) {
+ // The light index change path only publishes the new schema and submits index delete jobs. When a
+ // clause also needs a heavy schema change, the whole statement runs as a heavy schema change job,
+ // which applies the index changes as well.
+ boolean useLightIndexChange = Config.enable_light_index_change && lightIndexChange
+ && !requiresHeavyChange;
+ if (olapTable.hasChangelog() && !(lightSchemaChange || useLightIndexChange)) {
Review Comment:
Agreed that the in-place bloom-filter update survives this rejection, but it is the pre-existing class rather than something specific to this change: on master any statement on a changelog table that ends up on the heavy path is rejected by this same check after the clause loop, e.g. `DROP COLUMN v1, MODIFY COLUMN vi BIGINT NULL` leaves `v1` out of the bloom filter columns in the same way. This PR only routes one more statement shape there. Restoring the live metadata for every rejection before publishing (bloom filter columns, generated column references, sequence mapping groups) is what #69445 does, so I am keeping that out of this PR to keep the two changes independent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org