You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/10/23 03:37:31 UTC
[PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
morningman opened a new pull request, #69499:
URL: https://github.com/apache/doris/pull/69499
### What problem does this PR solve?
Issue Number: None
Related PR: #66538, #65851
Problem Summary:
**In short.** A write to an Iceberg branch binds to the schema of the branch's head snapshot. Iceberg, Spark and the Iceberg docs all bind a branch write to the table's current schema, and so do branch-4.1 and branch-4.2 (#66538). This PR does the same on master.
**Background.** #65851 pins a statement-scoped writer schema for every Iceberg write (`IcebergWriteSchemaContext`): the columns that the target list binds to, their write-defaults, and the partition spec and sort order bound to that schema. For `INSERT [OVERWRITE] INTO t@branch(b)` it pins the schema of the branch head snapshot, although a read of the branch already uses the table-current schema. Iceberg commits a branch write with the table-current schema: `SnapshotProducer` records `base.currentSchemaId()` as the schema of the new snapshot, whatever its target ref. So master also needs `validateBranchWriterSchema`, which checks that the current schema can describe files written with the branch head schema. The Iceberg docs ("Schema selection with branches and tags") say that a write to a branch uses the table schema and only a read of a tag uses the snapshot's schema. Spark 4.0 with Iceberg 1.10.1 binds branch writes the same way: after a rename on main, a write to an older bran
ch accepts the new name and rejects the old one. branch-4.1 and branch-4.2 (#66538) pin `table.schema()` for branch writes.
**The problem, and what it cost.** Reproduced on a local cluster with a `file://` format-version 3 table `t (id INT, c INT)` holding one row, with a branch `b` created before the change on main:
| Change on main after the branch | Statement | Master today | After this PR (same as Spark and branch-4.2) |
|---|---|---|---|
| `RENAME COLUMN c c2` | `INSERT INTO t@branch(b) (id, c2) VALUES (1, 1)` | `Unknown column 'c2' in target table` | writes the row |
| `RENAME COLUMN c c2` | `INSERT INTO t@branch(b) (id, c) VALUES (2, 2)` | writes the row | `Unknown column 'c' in target table` |
| `ADD COLUMN p INT`, then `ADD PARTITION KEY p` | `INSERT INTO t@branch(b) (id, c) VALUES (1, 1)` | `Iceberg partition field 1000 references source field 3 outside pinned schema 0` | writes the row, `p` is NULL |
| write-default of `default_int` changed from 35 to 36 (`test_iceberg_initial_defaults`, through the Iceberg API) | `INSERT OVERWRITE TABLE t@branch(b) (id, default_int) VALUES (14, DEFAULT)` | writes 35 | writes 36 |
The partition case has no way out: only a successful write to the branch moves its head to a newer schema, and every write to it fails.
**How this PR fixes it.**
- `IcebergWriteSchemaContext.create` still checks that the target ref exists and is a branch, not a tag. It then pins `table.schema()` for a branch, as for main, and binds the partition spec and sort order to that schema.
- `validateCurrentSchema`, the check before the commit, compares the pinned schema id with the table-current schema id for branch writes too. A schema change after the statement pinned its schema now fails the commit. Before, only a change of the branch head did.
- `validateBranchWriterSchema` is removed. The written files now carry the schema that Iceberg records for the new snapshot.
**Results.**
- Local cluster: the table above shows each statement before and after.
- Unit tests in `IcebergWritePlanProviderTest`: the five tests that pinned the branch head schema now expect the current schema. Together they cover the column names (also through `getWriteColumns` and the sink's schema JSON), the write-defaults, a current required column with and without a default (without one, an omitted value fails at analysis, as for main), and a schema change after pinning, which must fail the commit. Two new tests cover a current partition spec whose source column the branch head lacks, and a tag or a missing ref, which is still rejected. With master's `IcebergWriteSchemaContext`, 6 of these 7 tests fail and the tag test passes. All 1645 tests of `fe-connector-iceberg` pass.
- Regression: `test_iceberg_initial_defaults`, `write/test_iceberg_write_evolution_refs`, `branch_tag/iceberg_branch_tag_schema_change_extended` and `test_iceberg_schema_ref_actions_matrix` now expect what branch-4.2 expects. In `test_iceberg_initial_defaults` the branch write gets the current write-default 36. The other three write an older branch with the current column names and expect the old names to fail. All four pass locally. Without this change each one fails at the new expectation: the branch rows of `test_iceberg_initial_defaults` read 35, `test_iceberg_write_evolution_refs` and `iceberg_branch_tag_schema_change_extended` get `Unknown column 'zone'` and `Unknown column 'new_col'`, and in `test_iceberg_schema_ref_actions_matrix` the INSERT with the old column name succeeds.
### Release note
Fix: Writes to an Iceberg branch now use the table's current schema (column names, write-defaults and partition spec), as Spark does, instead of the schema of the branch's head snapshot.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
On a local cluster with a `file://` Iceberg catalog: `CREATE TABLE t (id INT, c INT) PROPERTIES ('format-version'='3')`, insert a row, `ALTER TABLE t CREATE BRANCH b`, then the changes and statements in the table above.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
See the release note. A write to a branch created before a schema change must use the current column names; the old names, which master accepted, now fail.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6367581049
<details>
<summary>ClickBench: <b>Total hot run time: 24.29 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2937f3442328f6ab64215bcaea1aa079c57af64c, data reload: false
query1 0.00 0.00 0.01
query2 0.10 0.05 0.04
query3 0.26 0.07 0.09
query4 1.61 0.10 0.10
query5 0.23 0.24 0.24
query6 1.17 0.68 0.68
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.40 0.36 0.35
query10 0.57 0.60 0.58
query11 0.20 0.15 0.14
query12 0.18 0.15 0.15
query13 0.48 0.48 0.48
query14 1.01 1.01 1.00
query15 0.61 0.60 0.61
query16 0.34 0.36 0.33
query17 1.11 1.13 1.13
query18 0.19 0.19 0.19
query19 2.11 2.00 1.98
query20 0.02 0.02 0.01
query21 15.52 0.19 0.14
query22 4.88 0.05 0.05
query23 16.28 0.32 0.12
query24 2.90 0.58 0.42
query25 0.11 0.07 0.07
query26 1.00 0.22 0.14
query27 0.06 0.03 0.05
query28 3.53 0.76 0.35
query29 12.50 4.15 3.26
query30 0.28 0.12 0.10
query31 2.78 0.55 0.33
query32 3.22 0.59 0.51
query33 3.14 3.25 3.15
query34 15.30 3.96 3.40
query35 3.36 3.35 3.34
query36 0.56 0.48 0.42
query37 0.10 0.06 0.07
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.16 0.13
query41 0.09 0.04 0.03
query42 0.04 0.02 0.03
query43 0.04 0.03 0.03
Total cold run time: 96.63 s
Total hot run time: 24.29 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6356788266
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6376183602
# FE UT Coverage Report
Increment line coverage ` 75.68% (28/37)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69499_49990114c9502537c6c096da0dd1eeeb8c833b6e/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69499_49990114c9502537c6c096da0dd1eeeb8c833b6e/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6375311199
<details>
<summary>ClickBench: <b>Total hot run time: 24.24 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 49990114c9502537c6c096da0dd1eeeb8c833b6e, data reload: false
query1 0.00 0.00 0.00
query2 0.10 0.05 0.05
query3 0.26 0.08 0.09
query4 1.61 0.09 0.10
query5 0.25 0.24 0.24
query6 1.17 0.69 0.68
query7 0.04 0.00 0.00
query8 0.06 0.04 0.04
query9 0.40 0.35 0.35
query10 0.58 0.61 0.62
query11 0.21 0.15 0.14
query12 0.17 0.17 0.15
query13 0.49 0.49 0.49
query14 1.05 1.02 1.01
query15 0.62 0.60 0.60
query16 0.31 0.35 0.36
query17 1.17 1.11 1.16
query18 0.21 0.19 0.19
query19 2.11 2.09 2.00
query20 0.02 0.01 0.01
query21 15.51 0.20 0.13
query22 4.87 0.06 0.05
query23 16.30 0.32 0.12
query24 2.94 0.57 0.40
query25 0.14 0.05 0.05
query26 1.24 0.21 0.17
query27 0.05 0.03 0.04
query28 4.25 0.80 0.37
query29 12.51 4.10 3.24
query30 0.28 0.13 0.10
query31 2.79 0.56 0.32
query32 3.22 0.59 0.50
query33 3.28 3.18 3.14
query34 15.30 3.99 3.36
query35 3.33 3.31 3.33
query36 0.57 0.45 0.41
query37 0.10 0.06 0.06
query38 0.05 0.03 0.04
query39 0.03 0.03 0.03
query40 0.18 0.15 0.14
query41 0.08 0.03 0.03
query42 0.03 0.04 0.03
query43 0.04 0.03 0.04
Total cold run time: 97.92 s
Total hot run time: 24.24 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6375228648
<details>
<summary>TPC-DS: <b>Total hot run time: 150114 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 49990114c9502537c6c096da0dd1eeeb8c833b6e, data reload: false
query5 4315 587 441 441
query6 417 201 196 196
query7 4809 449 221 221
query8 316 169 144 144
query9 8765 3061 3073 3061
query10 464 312 255 255
query11 6019 3861 3571 3571
query12 156 91 89 89
query13 1257 457 340 340
query14 6116 3920 3621 3621
query14_1 3424 3429 3371 3371
query15 212 199 180 180
query16 951 422 418 418
query17 877 668 539 539
query18 2430 421 305 305
query19 200 174 141 141
query20 89 83 82 82
query21 210 131 109 109
query22 12930 12990 12825 12825
query23 13498 12841 12503 12503
query23_1 12533 12577 12561 12561
query24 7365 827 464 464
query24_1 470 486 488 486
query25 553 420 365 365
query26 1292 251 142 142
query27 2774 445 277 277
query28 4597 1938 1936 1936
query29 1602 608 460 460
query30 301 222 186 186
query31 1018 942 815 815
query32 148 95 104 95
query33 532 301 254 254
query34 872 847 488 488
query35 740 758 644 644
query36 798 777 747 747
query37 118 104 85 85
query38 1778 1712 1655 1655
query39 802 779 753 753
query39_1 724 734 738 734
query40 221 123 95 95
query41 71 73 68 68
query42 90 86 85 85
query43 341 362 311 311
query44 1296 682 674 674
query45 179 196 210 196
query46 813 954 559 559
query47 2922 3049 2361 2361
query48 299 304 211 211
query49 575 397 296 296
query50 658 271 202 202
query51 10784 10790 10370 10370
query52 77 78 70 70
query53 201 218 154 154
query54 249 200 180 180
query55 77 70 64 64
query56 246 212 217 212
query57 1645 1612 1567 1567
query58 280 262 251 251
query59 2121 2238 2015 2015
query60 273 234 221 221
query61 142 145 143 143
query62 389 328 285 285
query63 185 156 153 153
query64 2810 941 774 774
query65 3344 3274 3254 3254
query66 1779 409 307 307
query67 20113 19825 19885 19825
query68 2989 991 592 592
query69 386 281 239 239
query70 824 748 757 748
query71 300 230 216 216
query72 2685 2577 2319 2319
query73 505 519 288 288
query74 4603 4503 4366 4366
query75 2351 2288 1909 1909
query76 2316 998 582 582
query77 335 385 275 275
query78 9784 9649 9099 9099
query79 989 837 508 508
query80 1232 427 331 331
query81 588 330 268 268
query82 621 145 104 104
query83 302 189 179 179
query84 308 124 104 104
query85 847 415 368 368
query86 393 237 229 229
query87 2015 1952 1794 1794
query88 3624 2636 2615 2615
query89 329 281 241 241
query90 1905 174 161 161
query91 164 145 124 124
query92 97 97 86 86
query93 1002 983 564 564
query94 636 317 264 264
query95 560 382 301 301
query96 650 479 235 235
query97 2469 2451 2345 2345
query98 162 151 152 151
query99 771 761 651 651
Total cold run time: 235173 ms
Total hot run time: 150114 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6365785398
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69499_2937f3442328f6ab64215bcaea1aa079c57af64c/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69499_2937f3442328f6ab64215bcaea1aa079c57af64c/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6367352164
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69499_2937f3442328f6ab64215bcaea1aa079c57af64c_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69499_2937f3442328f6ab64215bcaea1aa079c57af64c_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6356788697
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6361744956
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6367480641
<details>
<summary>TPC-DS: <b>Total hot run time: 150375 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2937f3442328f6ab64215bcaea1aa079c57af64c, data reload: false
query5 4304 595 435 435
query6 434 206 193 193
query7 4834 440 241 241
query8 320 163 150 150
query9 8772 3041 3113 3041
query10 486 305 257 257
query11 6013 3874 3560 3560
query12 154 93 90 90
query13 1278 448 329 329
query14 6139 3912 3584 3584
query14_1 3425 3368 3397 3368
query15 212 199 184 184
query16 957 415 406 406
query17 875 648 529 529
query18 2424 418 302 302
query19 205 165 136 136
query20 88 88 86 86
query21 214 127 109 109
query22 13070 12986 12716 12716
query23 13645 12961 12576 12576
query23_1 12612 12493 12562 12493
query24 7372 811 457 457
query24_1 483 465 474 465
query25 539 428 368 368
query26 1253 255 142 142
query27 2762 437 270 270
query28 4615 1929 1940 1929
query29 1578 603 456 456
query30 316 227 186 186
query31 997 936 817 817
query32 185 95 95 95
query33 529 314 255 255
query34 907 848 494 494
query35 749 752 636 636
query36 804 802 710 710
query37 126 98 84 84
query38 1815 1734 1679 1679
query39 799 802 732 732
query39_1 737 731 743 731
query40 241 136 109 109
query41 72 74 74 74
query42 89 86 86 86
query43 343 366 318 318
query44 1311 687 690 687
query45 182 179 179 179
query46 854 977 545 545
query47 2917 2776 2846 2776
query48 304 309 217 217
query49 575 412 283 283
query50 677 269 207 207
query51 10513 10522 10190 10190
query52 74 79 69 69
query53 189 210 149 149
query54 275 186 183 183
query55 73 66 62 62
query56 249 206 198 198
query57 1691 1603 1555 1555
query58 302 261 277 261
query59 2106 2163 1935 1935
query60 276 242 219 219
query61 145 142 147 142
query62 390 329 281 281
query63 186 157 158 157
query64 2784 938 744 744
query65 3314 3265 3248 3248
query66 1778 432 302 302
query67 20239 20164 20093 20093
query68 2981 1000 605 605
query69 407 285 240 240
query70 902 772 757 757
query71 299 236 215 215
query72 2754 2594 2329 2329
query73 523 534 291 291
query74 4624 4491 4327 4327
query75 2337 2282 1915 1915
query76 2308 1028 576 576
query77 340 368 268 268
query78 9814 9756 9285 9285
query79 1009 906 523 523
query80 835 428 340 340
query81 581 316 268 268
query82 823 132 102 102
query83 280 190 177 177
query84 322 122 104 104
query85 1008 426 373 373
query86 378 247 235 235
query87 2001 1957 1797 1797
query88 3702 2666 2657 2657
query89 356 286 240 240
query90 1722 168 170 168
query91 161 154 127 127
query92 97 88 88 88
query93 1034 1018 578 578
query94 558 300 270 270
query95 565 398 296 296
query96 671 477 234 234
query97 2456 2455 2302 2302
query98 158 155 145 145
query99 792 755 642 642
Total cold run time: 235886 ms
Total hot run time: 150375 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #69499:
URL: https://github.com/apache/doris/pull/69499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6360886926
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/178)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69499_a218feb8b0981efa6a17163b11fabe8d2b06fdd8_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69499_a218feb8b0981efa6a17163b11fabe8d2b06fdd8_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6356779271
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6361743697
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #69499:
URL: https://github.com/apache/doris/pull/69499#discussion_r4363772763
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriteSchemaContext.java:
##########
@@ -96,11 +94,12 @@ static IcebergWriteSchemaContext create(Table table, String tableName,
Objects.requireNonNull(table, "table should not be null");
Objects.requireNonNull(tableName, "tableName should not be null");
Objects.requireNonNull(branchName, "branchName should not be null");
- Schema schema = branchName.isPresent()
- ? resolveBranchSchema(table, branchName.get(), tableName) : table.schema();
if (branchName.isPresent()) {
- validateBranchWriterSchema(schema, table.schema(), branchName.get(), tableName);
+ validateTargetBranch(table, branchName.get(), tableName);
}
+ // A branch selects only the parent snapshot / ref. Iceberg validates a branch write against the
+ // table-current schema and stamps that schema on the new snapshot, and Spark binds it the same way.
+ Schema schema = table.schema();
Review Comment:
Fixed in 49990114c95. The interleaving is real, and it is not specific to branch writes: main-branch INSERT OVERWRITE on master has had the same exposure since #65851 (the outer pass pins the writer schema and then expands DEFAULT and bare rows by position against it) and #66529 (every attempt of the inner insert resets the statement scope and pins again). Before this PR a branch overwrite was exposed only when a concurrent commit moved the branch head onto a snapshot with another schema; binding branch writes to the table's current schema widened that to the same table-level changes as main.
The overwrite now records on the inner insert's `PluginDrivenInsertCommandContext` the writer columns and write metadata identity it normalized the plan against (`InsertUtils.recordNormalizedWriteGeneration`). After the inner insert pins again, `InsertUtils.checkNormalizedWriteGeneration` compares them (columns by position: name, field ID, type, nullability and default) and fails the statement with "retry the statement" when either differs, instead of binding the expanded values by position to the new schema. This covers main and branch overwrites alike, and an insert that normalizes its own plan is unaffected. branch-4.2 avoids the hole differently: it carries the pinned `IcebergWriteSchemaContext` on the unbound sink into the inner insert, so both passes share one generation, and its commit fence rejects a later change with "retry the statement". `InsertUtilsTest` replays the two pins with a reordered schema and with a changed identity (both fail without the check) and with an u
nchanged generation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6373750722
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6377480277
# FE Regression Coverage Report
Increment line coverage ` 67.57% (25/37)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69499_49990114c9502537c6c096da0dd1eeeb8c833b6e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69499_49990114c9502537c6c096da0dd1eeeb8c833b6e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6375071993
<details>
<summary>TPC-H: <b>Total hot run time: 28926 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 49990114c9502537c6c096da0dd1eeeb8c833b6e, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17623 5181 5074 5074
q2 2142 346 267 267
q3 10170 1429 827 827
q4 4683 476 339 339
q5 7471 809 524 524
q6 184 183 153 153
q7 747 803 606 606
q8 9257 1343 1155 1155
q9 5545 4232 4355 4232
q10 6813 1316 1027 1027
q11 440 254 235 235
q12 642 405 273 273
q13 18118 3655 3048 3048
q14 273 273 254 254
q15 q16 725 718 666 666
q17 1208 748 575 575
q18 7011 6202 6138 6138
q19 1190 964 605 605
q20 405 346 224 224
q21 5756 2496 2428 2428
q22 407 329 276 276
Total cold run time: 100810 ms
Total hot run time: 28926 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6009 5985 6197 5985
q2 722 589 582 582
q3 5472 5656 5283 5283
q4 2024 2171 1606 1606
q5 5844 5746 5786 5746
q6 239 184 141 141
q7 2260 2002 1756 1756
q8 3029 2699 2674 2674
q9 7991 7919 7864 7864
q10 3657 3596 3172 3172
q11 597 418 387 387
q12 653 681 506 506
q13 3304 3651 3058 3058
q14 289 313 275 275
q15 q16 683 676 625 625
q17 7819 7196 6899 6899
q18 12934 12203 12952 12203
q19 1057 799 825 799
q20 2140 2162 1915 1915
q21 5729 4761 4806 4761
q22 482 450 420 420
Total cold run time: 72934 ms
Total hot run time: 66657 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6367280925
<details>
<summary>TPC-H: <b>Total hot run time: 28692 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2937f3442328f6ab64215bcaea1aa079c57af64c, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17602 5170 5039 5039
q2 2177 313 252 252
q3 10227 1383 825 825
q4 4691 467 344 344
q5 7466 797 519 519
q6 186 180 153 153
q7 740 798 614 614
q8 9278 1294 1039 1039
q9 5527 4198 4178 4178
q10 6840 1301 1034 1034
q11 431 253 226 226
q12 641 403 276 276
q13 18090 4044 3032 3032
q14 270 274 253 253
q15 q16 762 728 674 674
q17 1208 735 577 577
q18 7035 6180 6099 6099
q19 1119 956 599 599
q20 383 340 228 228
q21 5612 2491 2434 2434
q22 392 335 297 297
Total cold run time: 100677 ms
Total hot run time: 28692 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5966 5909 6084 5909
q2 756 609 579 579
q3 5629 5773 5407 5407
q4 1988 2135 1564 1564
q5 5598 5533 5892 5533
q6 241 190 143 143
q7 2197 2042 1793 1793
q8 2954 2688 2678 2678
q9 7923 7912 7841 7841
q10 3608 3592 3133 3133
q11 572 406 376 376
q12 676 665 503 503
q13 3325 3603 3027 3027
q14 288 288 265 265
q15 q16 654 725 617 617
q17 7718 7165 6879 6879
q18 12978 12188 12929 12188
q19 886 778 775 775
q20 2132 2136 1933 1933
q21 5621 4555 4802 4555
q22 511 452 422 422
Total cold run time: 72221 ms
Total hot run time: 66120 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6361081137
<details>
<summary>TPC-DS: <b>Total hot run time: 150596 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a218feb8b0981efa6a17163b11fabe8d2b06fdd8, data reload: false
query5 4306 600 442 442
query6 434 203 197 197
query7 4803 442 224 224
query8 317 171 150 150
query9 8780 3067 3066 3066
query10 498 305 252 252
query11 5897 3851 3553 3553
query12 149 89 86 86
query13 1243 440 328 328
query14 6119 3875 3627 3627
query14_1 3372 3399 3364 3364
query15 209 194 190 190
query16 952 473 378 378
query17 860 666 531 531
query18 2431 428 313 313
query19 203 165 138 138
query20 89 82 84 82
query21 216 125 108 108
query22 12904 13127 12690 12690
query23 13341 13005 12537 12537
query23_1 12684 12553 12544 12544
query24 7390 827 468 468
query24_1 469 468 475 468
query25 557 415 370 370
query26 1262 262 141 141
query27 2768 440 278 278
query28 4629 1929 1884 1884
query29 1571 597 474 474
query30 299 219 188 188
query31 1012 931 814 814
query32 156 91 95 91
query33 539 305 247 247
query34 905 847 483 483
query35 739 739 637 637
query36 820 826 723 723
query37 121 98 84 84
query38 1776 1758 1674 1674
query39 790 811 747 747
query39_1 744 722 744 722
query40 235 121 97 97
query41 74 70 68 68
query42 92 86 82 82
query43 345 359 310 310
query44 1309 676 681 676
query45 213 181 165 165
query46 834 973 553 553
query47 2942 2935 2755 2755
query48 293 301 210 210
query49 568 392 284 284
query50 686 272 213 213
query51 10734 10529 10371 10371
query52 82 75 68 68
query53 185 206 162 162
query54 233 189 177 177
query55 73 70 63 63
query56 231 206 218 206
query57 1648 1600 1544 1544
query58 283 264 247 247
query59 2128 2091 1922 1922
query60 284 237 221 221
query61 147 150 148 148
query62 389 331 292 292
query63 185 156 165 156
query64 2755 939 772 772
query65 3328 3358 3261 3261
query66 1801 426 300 300
query67 20246 20167 20127 20127
query68 3285 997 614 614
query69 389 287 253 253
query70 832 776 788 776
query71 313 236 211 211
query72 2818 2596 2320 2320
query73 531 522 291 291
query74 4634 4536 4337 4337
query75 2339 2283 1903 1903
query76 2400 1000 583 583
query77 339 374 269 269
query78 9678 9722 9245 9245
query79 988 856 538 538
query80 1283 424 333 333
query81 605 319 274 274
query82 868 135 103 103
query83 301 194 185 185
query84 310 115 103 103
query85 876 427 388 388
query86 398 238 232 232
query87 1970 1935 1785 1785
query88 3652 2683 2642 2642
query89 330 292 247 247
query90 1951 167 157 157
query91 184 149 126 126
query92 96 79 83 79
query93 1005 1001 578 578
query94 653 303 265 265
query95 570 332 297 297
query96 658 489 231 231
query97 2474 2417 2322 2322
query98 162 152 145 145
query99 765 771 653 653
Total cold run time: 236116 ms
Total hot run time: 150596 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6373751384
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6360916429
<details>
<summary>TPC-H: <b>Total hot run time: 28362 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a218feb8b0981efa6a17163b11fabe8d2b06fdd8, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17619 5166 5057 5057
q2 2126 315 248 248
q3 10263 1389 810 810
q4 4682 469 331 331
q5 7428 791 530 530
q6 187 201 154 154
q7 743 788 588 588
q8 9239 1337 1065 1065
q9 5505 4187 4187 4187
q10 6819 1281 1055 1055
q11 427 249 224 224
q12 634 405 277 277
q13 18090 3570 3018 3018
q14 266 272 247 247
q15 q16 726 707 676 676
q17 1204 757 576 576
q18 7051 6172 6113 6113
q19 1107 950 588 588
q20 401 345 235 235
q21 5022 2134 2285 2134
q22 344 306 249 249
Total cold run time: 99883 ms
Total hot run time: 28362 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5640 5629 5605 5605
q2 667 573 510 510
q3 5195 5396 4882 4882
q4 2014 2141 1556 1556
q5 5278 5108 5556 5108
q6 262 207 159 159
q7 2332 2099 1892 1892
q8 3081 2868 2610 2610
q9 7977 8095 7940 7940
q10 3746 3710 3259 3259
q11 610 426 402 402
q12 682 687 514 514
q13 3558 3881 3368 3368
q14 322 316 294 294
q15 q16 681 709 657 657
q17 8305 7428 7349 7349
q18 13024 12267 12929 12267
q19 908 781 782 781
q20 2141 2139 1927 1927
q21 5698 4652 4896 4652
q22 501 452 414 414
Total cold run time: 72622 ms
Total hot run time: 66146 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6361164079
<details>
<summary>ClickBench: <b>Total hot run time: 24.39 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a218feb8b0981efa6a17163b11fabe8d2b06fdd8, data reload: false
query1 0.00 0.00 0.01
query2 0.09 0.05 0.05
query3 0.26 0.09 0.10
query4 1.61 0.12 0.10
query5 0.22 0.24 0.24
query6 1.17 0.68 0.68
query7 0.05 0.01 0.01
query8 0.05 0.04 0.04
query9 0.41 0.35 0.36
query10 0.57 0.58 0.56
query11 0.20 0.14 0.15
query12 0.18 0.14 0.15
query13 0.51 0.48 0.48
query14 1.05 1.01 1.02
query15 0.62 0.62 0.60
query16 0.35 0.35 0.36
query17 1.14 1.07 1.18
query18 0.22 0.19 0.20
query19 2.10 2.03 2.06
query20 0.02 0.02 0.01
query21 15.50 0.19 0.14
query22 4.89 0.05 0.04
query23 16.28 0.32 0.12
query24 2.84 0.65 0.41
query25 0.11 0.04 0.06
query26 1.46 0.22 0.15
query27 0.05 0.03 0.05
query28 3.67 0.82 0.38
query29 12.49 4.14 3.24
query30 0.30 0.12 0.10
query31 2.78 0.55 0.33
query32 3.23 0.58 0.50
query33 3.31 3.25 3.33
query34 15.30 4.00 3.38
query35 3.36 3.35 3.34
query36 0.55 0.49 0.44
query37 0.10 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.15
query41 0.08 0.03 0.02
query42 0.03 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 97.45 s
Total hot run time: 24.39 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69499:
URL: https://github.com/apache/doris/pull/69499#issuecomment-6359486485
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69499_a218feb8b0981efa6a17163b11fabe8d2b06fdd8/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69499_a218feb8b0981efa6a17163b11fabe8d2b06fdd8/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Bind Iceberg branch writes to the table's current schema [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69499:
URL: https://github.com/apache/doris/pull/69499#discussion_r4358174976
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriteSchemaContext.java:
##########
@@ -96,11 +94,12 @@ static IcebergWriteSchemaContext create(Table table, String tableName,
Objects.requireNonNull(table, "table should not be null");
Objects.requireNonNull(tableName, "tableName should not be null");
Objects.requireNonNull(branchName, "branchName should not be null");
- Schema schema = branchName.isPresent()
- ? resolveBranchSchema(table, branchName.get(), tableName) : table.schema();
if (branchName.isPresent()) {
- validateBranchWriterSchema(schema, table.schema(), branchName.get(), tableName);
+ validateTargetBranch(table, branchName.get(), tableName);
}
+ // A branch selects only the parent snapshot / ref. Iceberg validates a branch write against the
+ // table-current schema and stamps that schema on the new snapshot, and Spark binds it the same way.
+ Schema schema = table.schema();
Review Comment:
[P1] Preserve the writer generation across overwrite replanning
For a v3 table ordered `[a, b]` with write-defaults `1` and `2`, run `INSERT OVERWRITE t@branch(br) VALUES (DEFAULT, DEFAULT)`. `InsertOverwriteTableCommand.run` first pins that schema and `normalizePlan` produces `InlineTable(1, 2)`. If another Iceberg client commits `updateSchema().moveFirst("b").commit()` before the inner INSERT starts, `InsertIntoTableCommand.initPlan` calls `resetConnectorStatementScope()` and discards both the original schema and metadata identity. It then pins `[b, a]` while `needNormalizePlan=false` retains those already-expanded literals. `BindSink.getColumnToChildOutput` maps by ordinal, producing `Sink[b <- 1, a <- 2]`: the defaults are silently written into the wrong columns, and all schema checks pass against the second pin. Before this change, a table-only reorder left both branch-head schema pins unchanged. Preserve the outer writer context/identity across this handoff, or restart normalization from the original DEFAULT expressions after repinning;
cover the inter-pass reorder with a controlled concurrency test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org