You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/10/22 13:06:15 UTC
[PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
morningman opened a new pull request, #69471:
URL: https://github.com/apache/doris/pull/69471
### What problem does this PR solve?
Issue Number: None
Related PR: #65329, #64304
Problem Summary:
**In short.** This restores a rule that #65329 introduced and that branch-4.1 and branch-4.2 still follow: on an Iceberg format-version 3 table, `ALTER TABLE ... ADD COLUMN` and `RENAME COLUMN` may not name a top-level column `_row_id` or `_last_updated_sequence_number`. Master lost the ALTER half of the rule when #64304 moved Iceberg DDL into the connector plugin.
**Background.** Doris exposes the row lineage of a format-version 3 table as two hidden columns, `_row_id` and `_last_updated_sequence_number`, next to the table's own columns. #65329 (merged 07-23) rejected those names for the columns of a v3 table, in CREATE TABLE and in ALTER TABLE. #64304 (merged 07-30) was developed in parallel and rewrote Iceberg DDL in `fe-connector-iceberg`. It kept the CREATE TABLE check (`IcebergConnectorMetadata.rejectReservedRowLineageColumns`), but ALTER only checks `_file` and `_pos`. Iceberg itself does not check the names in a schema update.
**The problem, and what it cost.** On master these statements commit on a v3 table. Doris then appends the hidden column to a schema that already has a column with the same name. With the default metadata cache, `DESC` and every query on the table fail with `Duplicate column name found: _row_id`. With the table cache disabled, the user column hides the row lineage column instead. Spark cannot read such a table either: it fails with `Table column names conflict with names reserved for Iceberg metadata columns`.
| On a v3 table | Master today | After this PR (same as branch-4.2) |
|---|---|---|
| `ADD COLUMN _row_id BIGINT` | commits; the table can no longer be read | `Cannot add Iceberg v3 reserved row lineage column: _row_id` |
| `ADD COLUMN (extra INT, _LAST_UPDATED_SEQUENCE_NUMBER BIGINT)` | commits both columns | rejected; nothing is committed |
| `RENAME COLUMN c _last_updated_sequence_number` | commits | `Cannot rename to Iceberg v3 reserved row lineage column: _last_updated_sequence_number` |
| the same names on a v2 table, or on a nested field such as `s._row_id` | allowed | allowed |
**How this PR fixes it.** `CatalogBackedIcebergCatalogOps.addColumn`, `addColumns` and `renameColumn` call a new `IcebergWriterHelper.validateNotRowLineageColumn` after they load the table, next to the existing VARIANT format check, so the rule reads the table's real format version. The check ignores case and takes the names from Iceberg's `MetadataColumns`. The messages are the ones branch-4.2 uses.
One difference from branch-4.2 is deliberate. branch-4.2 also rejects dropping such a column or renaming it away. But Iceberg lets such a column exist on a v3 table: Spark adds one with `ADD COLUMN`, and a v2 table that has one can be upgraded to v3. After that neither Spark nor Doris can read the table, and dropping the column or renaming it away is how it becomes readable again. This PR keeps both allowed. `MODIFY COLUMN`, a column comment or a reorder that names `_row_id` already fails on master, because the row lineage columns are not in the Iceberg schema.
**Results.**
- Reproduced on a local cluster with a `file://` v3 table. Before: `ADD COLUMN _row_id BIGINT` and `RENAME COLUMN c _last_updated_sequence_number` commit, and `DESC` then fails with `Duplicate column name found: _row_id`. After: both fail with the messages above, and the table stays readable. A table that Spark upgraded from v2 with a `_row_id` column becomes readable in Doris and in Spark after `DROP COLUMN _row_id` in Doris, and so does one where Spark added the column, after a rename.
- New tests in `CatalogBackedIcebergCatalogOpsColumnEvolutionTest` run against a real in-memory Iceberg catalog. On a v3 table the three statements fail and leave the schema unchanged. v2 tables and nested fields may use the names. An existing column with such a name can be dropped or renamed away. The first test fails without this change. All 1640 tests of fe-connector-iceberg pass.
- `test_iceberg_v3_row_lineage_query_insert` now runs the three statements on its v3 table, for Parquet and ORC, and checks the errors. The rest of the suite then writes to and reads the same table. With the connector built without this change, the suite fails at the first of the three.
### Release note
Fix: On an Iceberg format-version 3 table, `ALTER TABLE ... ADD COLUMN` and `RENAME COLUMN` now fail when they would name a top-level column `_row_id` or `_last_updated_sequence_number`. Before, the statement succeeded and left the table unreadable.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
On a local cluster with a `file://` Iceberg catalog: `CREATE TABLE t (id INT, c STRING) PROPERTIES ('format-version'='3')`, then `ALTER TABLE t ADD COLUMN _row_id BIGINT` and `ALTER TABLE t RENAME COLUMN c _last_updated_sequence_number` fail and `DESC t` still works. A Spark table upgraded from v2 with a `_row_id` column becomes readable after `ALTER TABLE ... DROP COLUMN _row_id` in Doris.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
These ALTER statements now fail on format-version 3 tables. Dropping such a column or renaming it away stays allowed, unlike branch-4.2.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69471:
URL: https://github.com/apache/doris/pull/69471#issuecomment-6363010389
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69471:
URL: https://github.com/apache/doris/pull/69471#issuecomment-6348873654
<details>
<summary>TPC-H: <b>Total hot run time: 28540 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 68805b854c69886942f08f3689008ee431b48fdb, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17589 5204 5110 5110
q2 2107 320 270 270
q3 10231 1381 831 831
q4 4674 467 338 338
q5 7451 790 531 531
q6 178 183 152 152
q7 740 805 606 606
q8 9253 1329 1105 1105
q9 5507 4206 4190 4190
q10 6836 1282 1044 1044
q11 423 247 232 232
q12 646 404 290 290
q13 18102 3728 3000 3000
q14 278 277 243 243
q15 q16 733 720 684 684
q17 1221 758 579 579
q18 7053 6204 6146 6146
q19 1233 955 601 601
q20 374 329 224 224
q21 5488 2115 2295 2115
q22 356 299 249 249
Total cold run time: 100473 ms
Total hot run time: 28540 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5642 5635 5641 5635
q2 692 571 526 526
q3 5214 5379 4929 4929
q4 2019 2148 1548 1548
q5 5263 5162 5793 5162
q6 265 200 146 146
q7 2333 2144 1874 1874
q8 3203 2912 2647 2647
q9 8064 8037 8089 8037
q10 3745 3704 3236 3236
q11 633 440 439 439
q12 709 689 524 524
q13 3475 4013 3194 3194
q14 352 338 280 280
q15 q16 698 713 634 634
q17 8187 7470 7450 7450
q18 13047 12230 12944 12230
q19 889 780 795 780
q20 2162 2145 1934 1934
q21 5662 4620 4690 4620
q22 485 449 438 438
Total cold run time: 72739 ms
Total hot run time: 66263 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69471:
URL: https://github.com/apache/doris/pull/69471#issuecomment-6341506828
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69471:
URL: https://github.com/apache/doris/pull/69471#issuecomment-6349068967
<details>
<summary>TPC-DS: <b>Total hot run time: 150364 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 68805b854c69886942f08f3689008ee431b48fdb, data reload: false
query5 4306 572 434 434
query6 432 200 189 189
query7 4815 446 225 225
query8 321 168 143 143
query9 8774 3110 3065 3065
query10 483 299 258 258
query11 6037 3858 3596 3596
query12 155 93 89 89
query13 1268 451 316 316
query14 6138 3949 3605 3605
query14_1 3441 3426 3397 3397
query15 208 199 182 182
query16 945 425 412 412
query17 865 651 537 537
query18 2425 428 310 310
query19 210 162 144 144
query20 86 85 87 85
query21 219 137 105 105
query22 12970 13011 12823 12823
query23 13500 12791 12463 12463
query23_1 12581 12583 12472 12472
query24 7360 808 456 456
query24_1 466 472 478 472
query25 541 424 375 375
query26 1276 255 144 144
query27 2763 444 276 276
query28 4615 1961 1971 1961
query29 1562 608 446 446
query30 301 220 183 183
query31 1004 937 828 828
query32 139 93 92 92
query33 539 306 245 245
query34 902 857 488 488
query35 741 759 642 642
query36 818 828 725 725
query37 129 96 92 92
query38 1762 1745 1686 1686
query39 804 790 751 751
query39_1 734 743 735 735
query40 222 117 97 97
query41 72 70 69 69
query42 87 86 87 86
query43 345 375 310 310
query44 1313 673 687 673
query45 219 171 165 165
query46 812 959 567 567
query47 2935 2896 2768 2768
query48 289 301 210 210
query49 577 399 308 308
query50 681 270 214 214
query51 10283 10321 10264 10264
query52 79 76 72 72
query53 187 208 147 147
query54 242 192 186 186
query55 69 65 63 63
query56 221 227 199 199
query57 1635 1599 1566 1566
query58 281 263 269 263
query59 2089 2199 1925 1925
query60 274 243 219 219
query61 146 136 144 136
query62 388 325 281 281
query63 183 155 161 155
query64 2733 983 752 752
query65 3305 3283 3267 3267
query66 1765 422 292 292
query67 20229 20065 19960 19960
query68 3035 1004 606 606
query69 386 296 235 235
query70 831 775 758 758
query71 287 232 216 216
query72 2658 2597 2267 2267
query73 512 527 294 294
query74 4627 4506 4320 4320
query75 2338 2310 1884 1884
query76 2298 1006 585 585
query77 335 380 264 264
query78 9912 9707 9214 9214
query79 1003 849 513 513
query80 895 429 332 332
query81 573 319 265 265
query82 577 130 107 107
query83 296 207 185 185
query84 305 126 107 107
query85 937 413 372 372
query86 351 240 228 228
query87 1974 1974 1794 1794
query88 3606 2693 2660 2660
query89 327 285 240 240
query90 1755 163 154 154
query91 164 149 130 130
query92 99 88 90 88
query93 995 996 592 592
query94 505 308 272 272
query95 558 392 310 310
query96 665 480 234 234
query97 2447 2424 2313 2313
query98 161 153 155 153
query99 778 758 648 648
Total cold run time: 234072 ms
Total hot run time: 150364 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #69471:
URL: https://github.com/apache/doris/pull/69471
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69471:
URL: https://github.com/apache/doris/pull/69471#issuecomment-6341510442
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69471:
URL: https://github.com/apache/doris/pull/69471#issuecomment-6349172293
<details>
<summary>ClickBench: <b>Total hot run time: 24.32 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 68805b854c69886942f08f3689008ee431b48fdb, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.05
query3 0.26 0.10 0.09
query4 1.62 0.10 0.10
query5 0.26 0.24 0.24
query6 1.17 0.69 0.68
query7 0.04 0.01 0.00
query8 0.05 0.04 0.03
query9 0.41 0.37 0.35
query10 0.61 0.60 0.59
query11 0.20 0.15 0.15
query12 0.19 0.15 0.15
query13 0.49 0.49 0.50
query14 1.03 1.02 1.01
query15 0.63 0.62 0.60
query16 0.34 0.35 0.36
query17 1.13 1.11 1.12
query18 0.20 0.19 0.19
query19 2.14 1.95 2.03
query20 0.02 0.02 0.01
query21 15.43 0.19 0.13
query22 4.86 0.05 0.06
query23 16.26 0.32 0.14
query24 3.07 0.49 0.40
query25 0.12 0.06 0.06
query26 1.00 0.21 0.17
query27 0.05 0.05 0.05
query28 3.60 0.77 0.35
query29 12.50 4.22 3.23
query30 0.30 0.13 0.10
query31 2.76 0.56 0.31
query32 3.22 0.59 0.51
query33 3.23 3.28 3.20
query34 15.33 3.99 3.39
query35 3.33 3.35 3.31
query36 0.54 0.44 0.43
query37 0.09 0.07 0.06
query38 0.05 0.04 0.03
query39 0.03 0.04 0.03
query40 0.17 0.15 0.14
query41 0.09 0.03 0.03
query42 0.04 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 97 s
Total hot run time: 24.32 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69471:
URL: https://github.com/apache/doris/pull/69471#issuecomment-6363009667
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69471:
URL: https://github.com/apache/doris/pull/69471#issuecomment-6351639282
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69471_68805b854c69886942f08f3689008ee431b48fdb_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69471_68805b854c69886942f08f3689008ee431b48fdb_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject Iceberg v3 row lineage names in ALTER ADD and RENAME COLUMN [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69471:
URL: https://github.com/apache/doris/pull/69471#issuecomment-6341494816
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org