You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xylaaaaa (via GitHub)" <gi...@apache.org> on 2026/01/23 04:29:43 UTC
[PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
xylaaaaa opened a new pull request, #60169:
URL: https://github.com/apache/doris/pull/60169
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Currently, Doris does not support `ALTER TABLE ... MODIFY COLUMN` for complex type columns (STRUCT/ARRAY/MAP) in Iceberg external tables. This PR adds support for safe schema evolution of complex types in Iceberg tables.
**Supported operations:**
| Complex Type | Allowed Operations | Prohibited Operations |
|--------------|-------------------|----------------------|
| STRUCT | Append new fields, safe type promotion for existing fields | Field rename/delete |
| ARRAY | Element type safe promotion | Incompatible element type change |
| MAP | Value type safe promotion | Key type change |
**Safe type promotions supported in nested types:**
- `INT` → `BIGINT`, `LARGEINT`
- `TINYINT` → `SMALLINT`, `INT`, `BIGINT`, `LARGEINT`
- `SMALLINT` → `INT`, `BIGINT`, `LARGEINT`
- `BIGINT` → `LARGEINT`
- `FLOAT` → `DOUBLE`
- `VARCHAR(n)` → `VARCHAR(m)` where m > n
**Constraints:**
- All new nested fields must be nullable
- Cannot change optional to required
- Complex type default values only support NULL
### Release note
Support ALTER TABLE MODIFY COLUMN for complex types (STRUCT/ARRAY/MAP) in Iceberg external tables, including appending struct fields and safe type promotions.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
**Manual test steps:**
```sql
-- Create Iceberg table with complex types
CREATE TABLE iceberg_catalog.test_db.iceberg_tbl (
id BIGINT,
user_info STRUCT<name:STRING, scores:ARRAY<INT>, age:INT>,
dt STRING
);
-- Test 1: STRUCT append field (should succeed)
ALTER TABLE iceberg_catalog.test_db.iceberg_tbl
MODIFY COLUMN user_info STRUCT<name:STRING, scores:ARRAY<INT>, age:INT, email:STRING>;
-- Test 2: STRUCT nested ARRAY element type promotion (should succeed)
ALTER TABLE iceberg_catalog.test_db.iceberg_tbl
MODIFY COLUMN user_info STRUCT<name:STRING, scores:ARRAY<BIGINT>, age:INT, email:STRING>;
-- Test 3: STRUCT field rename (should fail)
ALTER TABLE iceberg_catalog.test_db.iceberg_tbl
MODIFY COLUMN user_info STRUCT<full_name:STRING, scores:ARRAY<BIGINT>, age:INT, email:STRING>;
-- Expected error: Cannot rename struct field from 'name' to 'full_name'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3839479850
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3876592253
# FE Regression Coverage Report
Increment line coverage ` 4.62% (6/130)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60169_5fdf9f7dcc70ed8e638d525151b02e63e7fee193_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60169_5fdf9f7dcc70ed8e638d525151b02e63e7fee193_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3788158271
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3985392680
# FE UT Coverage Report
Increment line coverage ` 19.15% (27/141)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60169_224005d6e326c7112c8081de7fba7badb24ec073/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60169_224005d6e326c7112c8081de7fba7badb24ec073/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3839831318
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3985209190
<details>
<summary>TPC-DS: <b>Total hot run time: 183872 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 224005d6e326c7112c8081de7fba7badb24ec073, data reload: false
query5 4814 646 502 502
query6 357 213 198 198
query7 4222 462 265 265
query8 352 259 258 258
query9 8798 2768 2715 2715
query10 525 405 349 349
query11 16990 17512 17261 17261
query12 204 135 127 127
query13 1287 503 362 362
query14 6763 3458 3043 3043
query14_1 3005 2923 2897 2897
query15 216 211 182 182
query16 1044 516 491 491
query17 1351 861 624 624
query18 2715 518 383 383
query19 509 214 185 185
query20 136 131 143 131
query21 219 140 120 120
query22 5514 5239 4880 4880
query23 17198 16851 16536 16536
query23_1 16761 16720 16571 16571
query24 7386 1624 1218 1218
query24_1 1239 1252 1227 1227
query25 543 454 402 402
query26 1245 264 160 160
query27 2739 506 294 294
query28 4417 1888 1902 1888
query29 805 568 465 465
query30 315 250 206 206
query31 883 729 649 649
query32 81 72 69 69
query33 518 344 285 285
query34 899 905 568 568
query35 618 680 598 598
query36 1086 1077 968 968
query37 129 94 86 86
query38 3005 2973 2910 2910
query39 907 889 836 836
query39_1 839 831 822 822
query40 231 148 137 137
query41 62 60 60 60
query42 106 104 106 104
query43 369 387 352 352
query44
query45 200 191 187 187
query46 882 991 617 617
query47 2105 2160 2018 2018
query48 310 321 227 227
query49 616 470 387 387
query50 710 281 221 221
query51 4100 4089 4044 4044
query52 106 106 97 97
query53 297 338 286 286
query54 296 278 275 275
query55 89 85 81 81
query56 309 346 326 326
query57 1358 1322 1276 1276
query58 287 282 273 273
query59 2654 2739 2611 2611
query60 351 336 336 336
query61 153 145 144 144
query62 621 585 552 552
query63 312 277 275 275
query64 4893 1276 989 989
query65
query66 1408 479 374 374
query67 16405 16372 16390 16372
query68
query69 418 313 294 294
query70 1006 1016 920 920
query71 357 327 304 304
query72 3033 2744 2449 2449
query73 547 560 330 330
query74 9988 9937 9779 9779
query75 2850 2735 2480 2480
query76 2305 1022 679 679
query77 363 397 316 316
query78 11232 11471 10664 10664
query79 1533 834 624 624
query80 1360 622 526 526
query81 567 284 250 250
query82 985 151 116 116
query83 343 262 247 247
query84 255 118 102 102
query85 881 497 428 428
query86 430 295 285 285
query87 3132 3183 2989 2989
query88 3620 2670 2657 2657
query89 427 367 347 347
query90 2006 188 182 182
query91 165 154 139 139
query92 77 79 74 74
query93 1020 846 520 520
query94 649 322 288 288
query95 590 329 315 315
query96 651 521 230 230
query97 2514 2488 2396 2396
query98 227 216 232 216
query99 1009 992 928 928
Total cold run time: 256096 ms
Total hot run time: 183872 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3990047494
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3867547634
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3867779759
<details>
<summary>TPC-H: <b>Total hot run time: 30899 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f0a58c95498bb1dacde266c9642f7d4fa2904c18, data reload: false
------ Round 1 ----------------------------------
q1 17602 4537 4305 4305
q2 2034 349 261 261
q3 10122 1307 747 747
q4 10200 784 317 317
q5 7514 2181 1910 1910
q6 200 176 146 146
q7 900 735 622 622
q8 9272 1383 1141 1141
q9 4769 4604 4609 4604
q10 6832 1933 1547 1547
q11 497 322 300 300
q12 375 380 235 235
q13 17767 4043 3245 3245
q14 232 251 215 215
q15 872 796 814 796
q16 689 683 623 623
q17 687 771 620 620
q18 6576 5916 6277 5916
q19 1658 1063 663 663
q20 564 557 411 411
q21 2900 1980 1991 1980
q22 389 330 295 295
Total cold run time: 102651 ms
Total hot run time: 30899 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4586 4605 4625 4605
q2 272 344 257 257
q3 2286 2902 2434 2434
q4 1530 1909 1395 1395
q5 4721 4472 4456 4456
q6 215 170 139 139
q7 2032 2181 1735 1735
q8 2584 2478 2506 2478
q9 7432 7558 7517 7517
q10 2866 3038 2582 2582
q11 568 458 453 453
q12 663 678 551 551
q13 3548 4066 3242 3242
q14 267 281 265 265
q15 820 794 784 784
q16 629 675 631 631
q17 1089 1262 1311 1262
q18 7733 7427 7423 7423
q19 888 831 841 831
q20 1962 2054 1864 1864
q21 4567 4154 4025 4025
q22 567 554 518 518
Total cold run time: 51825 ms
Total hot run time: 49447 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3990047172
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3839998587
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3985144743
<details>
<summary>TPC-H: <b>Total hot run time: 28922 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 224005d6e326c7112c8081de7fba7badb24ec073, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17635 4538 4308 4308
q2 q3 10650 778 530 530
q4 4681 356 261 261
q5 7551 1196 1038 1038
q6 179 173 151 151
q7 777 847 670 670
q8 9379 1507 1425 1425
q9 5234 4730 4644 4644
q10 6863 1893 1646 1646
q11 473 267 238 238
q12 757 576 474 474
q13 17782 4221 3408 3408
q14 231 231 224 224
q15 989 800 787 787
q16 766 718 663 663
q17 719 879 426 426
q18 5899 5520 5281 5281
q19 1221 977 648 648
q20 524 509 400 400
q21 4568 1895 1455 1455
q22 338 286 245 245
Total cold run time: 97216 ms
Total hot run time: 28922 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4493 4346 4337 4337
q2 q3 1765 2183 1712 1712
q4 870 1146 771 771
q5 4031 4347 4290 4290
q6 179 174 146 146
q7 1749 1608 1495 1495
q8 2417 2640 2511 2511
q9 7629 7472 7369 7369
q10 2672 2952 2405 2405
q11 506 460 445 445
q12 531 584 471 471
q13 4019 4406 3744 3744
q14 285 319 284 284
q15 835 808 800 800
q16 734 757 717 717
q17 1162 1546 1314 1314
q18 7092 6868 6578 6578
q19 1005 969 918 918
q20 2123 2178 2095 2095
q21 4202 3610 3338 3338
q22 477 426 378 378
Total cold run time: 48776 ms
Total hot run time: 46118 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #60169:
URL: https://github.com/apache/doris/pull/60169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3984933634
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3840435190
<details>
<summary>TPC-H: <b>Total hot run time: 32175 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f0a58c95498bb1dacde266c9642f7d4fa2904c18, data reload: false
------ Round 1 ----------------------------------
q1 17708 5254 5069 5069
q2 2002 308 194 194
q3 10242 1306 771 771
q4 10217 880 319 319
q5 7505 2172 1943 1943
q6 200 185 149 149
q7 912 744 612 612
q8 9273 1469 1144 1144
q9 5332 4829 4867 4829
q10 6870 1957 1569 1569
q11 508 312 294 294
q12 374 376 226 226
q13 17782 4095 3280 3280
q14 240 235 234 234
q15 925 817 825 817
q16 683 680 615 615
q17 646 776 512 512
q18 6889 6701 6496 6496
q19 1142 1152 614 614
q20 398 344 238 238
q21 2684 2015 1980 1980
q22 360 315 270 270
Total cold run time: 102892 ms
Total hot run time: 32175 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5310 5312 5300 5300
q2 262 349 264 264
q3 2190 2658 2241 2241
q4 1362 1756 1313 1313
q5 4295 4234 4269 4234
q6 222 185 142 142
q7 2363 2142 1846 1846
q8 2574 2482 2441 2441
q9 7582 7510 7687 7510
q10 3047 3008 2634 2634
q11 554 483 470 470
q12 728 821 680 680
q13 3872 4384 3459 3459
q14 294 323 301 301
q15 898 823 841 823
q16 677 711 676 676
q17 1185 1355 1368 1355
q18 7877 8091 7988 7988
q19 881 867 851 851
q20 2081 2123 2081 2081
q21 4883 4534 4463 4463
q22 553 545 526 526
Total cold run time: 53690 ms
Total hot run time: 51598 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3868218109
# FE Regression Coverage Report
Increment line coverage ` 4.62% (6/130)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60169_f0a58c95498bb1dacde266c9642f7d4fa2904c18_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60169_f0a58c95498bb1dacde266c9642f7d4fa2904c18_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3868238961
# FE Regression Coverage Report
Increment line coverage ` 4.62% (6/130)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60169_f0a58c95498bb1dacde266c9642f7d4fa2904c18_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60169_f0a58c95498bb1dacde266c9642f7d4fa2904c18_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3875919799
<details>
<summary>TPC-DS: <b>Total hot run time: 190512 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5fdf9f7dcc70ed8e638d525151b02e63e7fee193, data reload: false
query5 4316 621 495 495
query6 347 207 214 207
query7 4216 466 268 268
query8 351 252 243 243
query9 8729 2759 2792 2759
query10 518 369 321 321
query11 16376 16547 16159 16159
query12 189 121 119 119
query13 1274 472 350 350
query14 6149 3229 2986 2986
query14_1 2822 2748 2738 2738
query15 203 189 170 170
query16 971 376 431 376
query17 1063 677 562 562
query18 2438 423 331 331
query19 195 204 180 180
query20 134 127 120 120
query21 213 145 123 123
query22 4823 4848 4897 4848
query23 19065 18622 18257 18257
query23_1 18448 18349 18359 18349
query24 7089 1578 1230 1230
query24_1 1206 1228 1235 1228
query25 524 430 386 386
query26 1231 268 153 153
query27 2780 459 301 301
query28 4505 1867 1876 1867
query29 798 559 469 469
query30 326 257 220 220
query31 892 748 655 655
query32 92 80 81 80
query33 549 368 301 301
query34 910 916 578 578
query35 648 680 612 612
query36 1098 1082 979 979
query37 139 99 91 91
query38 3024 2916 2883 2883
query39 954 929 900 900
query39_1 895 895 882 882
query40 225 138 120 120
query41 73 68 68 68
query42 110 102 101 101
query43 424 435 415 415
query44 1329 739 744 739
query45 213 192 186 186
query46 879 1000 598 598
query47 2184 2180 2053 2053
query48 320 333 242 242
query49 638 422 355 355
query50 664 267 224 224
query51 4060 4133 4031 4031
query52 102 101 92 92
query53 286 321 264 264
query54 288 268 250 250
query55 94 87 79 79
query56 312 307 295 295
query57 1398 1352 1241 1241
query58 280 258 252 252
query59 2004 2065 2132 2065
query60 339 335 321 321
query61 148 144 144 144
query62 606 580 496 496
query63 297 262 277 262
query64 4796 1233 919 919
query65 4534 4429 4397 4397
query66 1488 451 331 331
query67 16282 16207 16107 16107
query68 2421 1063 713 713
query69 396 327 272 272
query70 1031 963 959 959
query71 328 312 293 293
query72 2865 2790 2592 2592
query73 509 550 312 312
query74 9693 9654 9499 9499
query75 2805 2783 2456 2456
query76 2286 1042 676 676
query77 352 366 305 305
query78 11299 11372 10874 10874
query79 1066 919 590 590
query80 642 573 512 512
query81 494 282 263 263
query82 1372 150 114 114
query83 362 258 240 240
query84 250 120 90 90
query85 828 480 424 424
query86 366 330 290 290
query87 3174 3133 2996 2996
query88 3554 2699 2668 2668
query89 394 338 329 329
query90 1874 185 173 173
query91 167 155 129 129
query92 78 77 70 70
query93 886 860 481 481
query94 431 308 300 300
query95 580 409 321 321
query96 645 511 230 230
query97 2480 2497 2401 2401
query98 228 211 216 211
query99 950 917 829 829
Total cold run time: 260988 ms
Total hot run time: 190512 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3875941337
<details>
<summary>ClickBench: <b>Total hot run time: 28.26 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5fdf9f7dcc70ed8e638d525151b02e63e7fee193, data reload: false
query1 0.05 0.05 0.06
query2 0.09 0.05 0.04
query3 0.25 0.09 0.09
query4 1.61 0.12 0.10
query5 0.26 0.24 0.26
query6 1.16 0.68 0.67
query7 0.03 0.02 0.03
query8 0.05 0.04 0.04
query9 0.55 0.50 0.49
query10 0.54 0.55 0.56
query11 0.14 0.09 0.09
query12 0.14 0.11 0.10
query13 0.64 0.62 0.61
query14 1.05 1.06 1.05
query15 0.87 0.87 0.89
query16 0.38 0.39 0.40
query17 1.14 1.10 1.14
query18 0.23 0.22 0.21
query19 2.08 2.02 2.02
query20 0.02 0.02 0.01
query21 15.43 0.27 0.15
query22 5.06 0.05 0.05
query23 15.84 0.28 0.11
query24 1.12 0.30 0.19
query25 0.09 0.06 0.07
query26 0.14 0.14 0.13
query27 0.06 0.07 0.05
query28 3.35 1.15 0.98
query29 12.54 3.97 3.18
query30 0.28 0.13 0.11
query31 2.80 0.63 0.42
query32 3.25 0.60 0.49
query33 3.19 3.25 3.32
query34 16.37 5.41 4.68
query35 4.79 4.80 4.84
query36 0.64 0.49 0.49
query37 0.11 0.07 0.07
query38 0.07 0.04 0.04
query39 0.05 0.03 0.03
query40 0.20 0.16 0.16
query41 0.09 0.03 0.02
query42 0.05 0.03 0.04
query43 0.04 0.04 0.03
Total cold run time: 96.84 s
Total hot run time: 28.26 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3867812891
<details>
<summary>ClickBench: <b>Total hot run time: 28.24 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f0a58c95498bb1dacde266c9642f7d4fa2904c18, data reload: false
query1 0.06 0.04 0.04
query2 0.10 0.05 0.05
query3 0.26 0.09 0.08
query4 1.61 0.12 0.11
query5 0.27 0.26 0.24
query6 1.16 0.68 0.67
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.57 0.51 0.50
query10 0.55 0.55 0.55
query11 0.14 0.09 0.10
query12 0.14 0.10 0.11
query13 0.64 0.62 0.62
query14 1.05 1.06 1.06
query15 0.88 0.85 0.86
query16 0.40 0.41 0.39
query17 1.17 1.07 1.09
query18 0.22 0.22 0.21
query19 2.09 2.01 2.06
query20 0.02 0.01 0.02
query21 15.39 0.28 0.14
query22 5.32 0.06 0.05
query23 16.05 0.30 0.10
query24 1.70 0.58 0.19
query25 0.06 0.06 0.08
query26 0.16 0.14 0.14
query27 0.08 0.06 0.08
query28 3.95 1.17 0.97
query29 12.56 3.91 3.16
query30 0.27 0.13 0.12
query31 2.82 0.66 0.40
query32 3.24 0.59 0.52
query33 3.29 3.35 3.28
query34 16.26 5.51 4.71
query35 4.76 4.76 4.80
query36 0.64 0.50 0.48
query37 0.11 0.07 0.07
query38 0.08 0.05 0.05
query39 0.05 0.03 0.03
query40 0.20 0.16 0.15
query41 0.09 0.04 0.03
query42 0.04 0.03 0.02
query43 0.04 0.04 0.04
Total cold run time: 98.56 s
Total hot run time: 28.24 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3875851408
<details>
<summary>TPC-H: <b>Total hot run time: 30461 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5fdf9f7dcc70ed8e638d525151b02e63e7fee193, data reload: false
------ Round 1 ----------------------------------
q1 17597 4547 4280 4280
q2 2078 356 227 227
q3 10110 1287 736 736
q4 10194 781 313 313
q5 7544 2197 1885 1885
q6 194 177 147 147
q7 905 744 601 601
q8 9277 1427 1077 1077
q9 4977 4611 4581 4581
q10 6861 1920 1527 1527
q11 497 294 291 291
q12 348 377 225 225
q13 17774 4024 3238 3238
q14 240 232 212 212
q15 875 806 798 798
q16 683 676 624 624
q17 704 769 576 576
q18 6897 5772 6128 5772
q19 1403 1100 687 687
q20 562 631 422 422
q21 2832 1941 2000 1941
q22 400 335 301 301
Total cold run time: 102952 ms
Total hot run time: 30461 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4537 4698 4514 4514
q2 271 353 265 265
q3 2301 2823 2442 2442
q4 1551 1837 1402 1402
q5 4803 4794 4549 4549
q6 216 178 137 137
q7 1953 1904 1746 1746
q8 2511 2451 2371 2371
q9 7500 7786 7547 7547
q10 2768 2954 2771 2771
q11 542 480 451 451
q12 659 735 590 590
q13 3886 4041 3264 3264
q14 269 299 262 262
q15 830 773 773 773
q16 639 675 641 641
q17 1078 1251 1276 1251
q18 7659 7464 7356 7356
q19 858 814 828 814
q20 1941 2076 1941 1941
q21 4542 4319 4134 4134
q22 585 548 490 490
Total cold run time: 51899 ms
Total hot run time: 49711 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3876007321
# FE UT Coverage Report
Increment line coverage ` 15.38% (20/130)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60169_5fdf9f7dcc70ed8e638d525151b02e63e7fee193/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60169_5fdf9f7dcc70ed8e638d525151b02e63e7fee193/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3868229999
# FE Regression Coverage Report
Increment line coverage ` 4.62% (6/130)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60169_f0a58c95498bb1dacde266c9642f7d4fa2904c18_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60169_f0a58c95498bb1dacde266c9642f7d4fa2904c18_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3841033873
# FE Regression Coverage Report
Increment line coverage ` 4.62% (6/130)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60169_f0a58c95498bb1dacde266c9642f7d4fa2904c18_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60169_f0a58c95498bb1dacde266c9642f7d4fa2904c18_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3984022396
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #60169:
URL: https://github.com/apache/doris/pull/60169#discussion_r2719570518
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java:
##########
@@ -716,15 +723,36 @@ public void renameColumn(ExternalTable dorisTable, String oldName, String newNam
public void modifyColumn(ExternalTable dorisTable, Column column, ColumnPosition position, long updateTime)
throws UserException {
Table icebergTable = IcebergUtils.getIcebergTable(dorisTable);
- validateForModifyColumn(column, icebergTable);
- Type icebergType = IcebergUtils.dorisTypeToIcebergType(column.getType());
+ NestedField currentCol = icebergTable.schema().findField(column.getName());
+ if (currentCol == null) {
+ throw new UserException("Column " + column.getName() + " does not exist");
+ }
+
+ validateCommonColumnInfo(column);
UpdateSchema updateSchema = icebergTable.updateSchema();
- updateSchema.updateColumn(column.getName(), icebergType.asPrimitiveType(), column.getComment());
- if (column.isAllowNull()) {
- // we can change a required column to optional, but not the other way around
- // because we don't know whether there is existing data with null values.
- updateSchema.makeColumnOptional(column.getName());
+
+ if (column.getType().isComplexType()) {
+ // Complex type processing branch
+ validateForModifyComplexColumn(column, currentCol);
+ applyComplexTypeChange(updateSchema, column.getName(), currentCol.type(), column.getType());
+ if (column.isAllowNull()) {
+ updateSchema.makeColumnOptional(column.getName());
+ }
+ if (!Objects.equals(currentCol.doc(), column.getComment())) {
+ updateSchema.updateColumnDoc(column.getName(), column.getComment());
+ }
+ } else {
Review Comment:
The new complex-type schema evolution logic in `modifyColumn` (STRUCT/ARRAY/MAP handling, nested promotions, nullability checks, etc.) isn’t covered by any tests in `IcebergMetadataOpTest` or other Iceberg tests, so regressions here would be hard to catch. Given that the rest of the Iceberg integration has unit tests, please consider adding tests that exercise successful and failing `ALTER TABLE ... MODIFY COLUMN` cases for complex types (e.g., struct field append, nested array element promotions, map value promotions, illegal field renames).
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java:
##########
@@ -737,23 +765,162 @@ public void modifyColumn(ExternalTable dorisTable, Column column, ColumnPosition
refreshTable(dorisTable, updateTime);
}
- private void validateForModifyColumn(Column column, Table icebergTable) throws UserException {
- validateCommonColumnInfo(column);
+ private void validateForModifyColumn(Column column, NestedField currentCol) throws UserException {
// check complex type
if (column.getType().isComplexType()) {
throw new UserException("Modify column type to non-primitive type is not supported: " + column.getType());
}
- // check exist
- NestedField currentCol = icebergTable.schema().findField(column.getName());
- if (currentCol == null) {
- throw new UserException("Column " + column.getName() + " does not exist");
- }
// check nullable
if (currentCol.isOptional() && !column.isAllowNull()) {
throw new UserException("Can not change nullable column " + column.getName() + " to not null");
}
}
+ private void validateForModifyComplexColumn(Column column, NestedField currentCol) throws UserException {
+ if (!column.getType().isComplexType()) {
+ throw new UserException("Modify column type to non-complex type is not supported: " + column.getType());
+ }
+ org.apache.doris.catalog.Type oldDorisType = IcebergUtils.icebergTypeToDorisType(
+ currentCol.type(), false, false);
+ org.apache.doris.catalog.Type newDorisType = column.getType();
+ try {
+ ColumnType.checkSupportSchemaChangeForComplexType(oldDorisType, newDorisType, false);
+ } catch (DdlException e) {
+ throw new UserException(e.getMessage(), e);
+ }
+ if (currentCol.isOptional() && !column.isAllowNull()) {
+ throw new UserException("Cannot change nullable column " + column.getName() + " to not null");
+ }
+ if (column.getDefaultValue() != null || column.getDefaultValueExprDef() != null) {
+ throw new UserException("Complex type default value only supports NULL");
+ }
+ }
+
+ private void applyComplexTypeChange(UpdateSchema updateSchema, String path,
+ org.apache.iceberg.types.Type oldIcebergType,
+ org.apache.doris.catalog.Type newDorisType) throws UserException {
+ switch (oldIcebergType.typeId()) {
+ case STRUCT:
+ applyStructChange(updateSchema, path, oldIcebergType.asStructType(), (StructType) newDorisType);
+ break;
+ case LIST:
+ applyListChange(updateSchema, path, (Types.ListType) oldIcebergType, (ArrayType) newDorisType);
+ break;
+ case MAP:
Review Comment:
`applyComplexTypeChange` decides how to cast `newDorisType` based on `oldIcebergType.typeId()`, but there is no validation that the new type has the same complex kind (STRUCT/ARRAY/MAP) as the existing column. For example, altering a STRUCT column to ARRAY (or a primitive column to a complex type) will pass `validateForModifyComplexColumn` but then hit a `ClassCastException` when casting `newDorisType` to `StructType`/`ArrayType`/`MapType`, resulting in an internal error instead of a user-facing validation error. It would be safer to explicitly check in `validateForModifyComplexColumn` (or here) that the old and new Doris types are both complex and of the same kind, and throw a `UserException` when they differ, before calling this method.
```suggestion
org.apache.doris.catalog.Type newDorisType) throws UserException {
// Ensure the new Doris type is complex before performing any casts based on the old Iceberg type.
if (!newDorisType.isComplexType()) {
throw new UserException("Modify column type to non-complex type is not supported: " + newDorisType);
}
switch (oldIcebergType.typeId()) {
case STRUCT:
if (!(newDorisType instanceof StructType)) {
throw new UserException("Cannot modify STRUCT column to type: " + newDorisType);
}
applyStructChange(updateSchema, path, oldIcebergType.asStructType(), (StructType) newDorisType);
break;
case LIST:
if (!(newDorisType instanceof ArrayType)) {
throw new UserException("Cannot modify LIST/ARRAY column to type: " + newDorisType);
}
applyListChange(updateSchema, path, (Types.ListType) oldIcebergType, (ArrayType) newDorisType);
break;
case MAP:
if (!(newDorisType instanceof MapType)) {
throw new UserException("Cannot modify MAP column to type: " + newDorisType);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3840525225
<details>
<summary>ClickBench: <b>Total hot run time: 28.5 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f0a58c95498bb1dacde266c9642f7d4fa2904c18, data reload: false
query1 0.06 0.04 0.04
query2 0.10 0.05 0.04
query3 0.25 0.08 0.08
query4 1.61 0.12 0.11
query5 0.27 0.24 0.25
query6 1.19 0.68 0.67
query7 0.03 0.02 0.02
query8 0.05 0.04 0.04
query9 0.56 0.50 0.49
query10 0.56 0.56 0.56
query11 0.14 0.09 0.10
query12 0.14 0.10 0.11
query13 0.63 0.61 0.63
query14 1.06 1.04 1.04
query15 0.88 0.86 0.87
query16 0.38 0.38 0.38
query17 1.12 1.12 1.16
query18 0.23 0.20 0.21
query19 2.04 2.03 2.07
query20 0.03 0.01 0.01
query21 15.40 0.26 0.14
query22 5.15 0.05 0.04
query23 15.98 0.28 0.11
query24 1.70 0.64 0.41
query25 0.12 0.11 0.05
query26 0.15 0.14 0.13
query27 0.09 0.05 0.07
query28 4.65 1.13 0.96
query29 12.56 3.90 3.19
query30 0.28 0.13 0.12
query31 2.82 0.65 0.41
query32 3.24 0.61 0.50
query33 3.31 3.32 3.26
query34 16.45 5.32 4.73
query35 4.81 4.84 4.81
query36 0.65 0.50 0.49
query37 0.11 0.07 0.07
query38 0.07 0.04 0.03
query39 0.04 0.03 0.02
query40 0.19 0.18 0.16
query41 0.08 0.03 0.03
query42 0.05 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 99.27 s
Total hot run time: 28.5 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3875731683
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](iceberg) Support schema change for complex types in Iceberg external tables [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60169:
URL: https://github.com/apache/doris/pull/60169#issuecomment-3868195835
# FE Regression Coverage Report
Increment line coverage ` 4.62% (6/130)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60169_f0a58c95498bb1dacde266c9642f7d4fa2904c18_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60169_f0a58c95498bb1dacde266c9642f7d4fa2904c18_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org