You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2026/10/16 02:42:34 UTC
[PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Gabriel39 opened a new pull request, #69158:
URL: https://github.com/apache/doris/pull/69158
### What problem does this PR solve?
A normal SELECT through a Paimon REST catalog can fail with:
```
Paimon table generation changed after the statement was pinned; retry the statement
```
For example, `test_paimon_rest_catalog_write` creates a table and completes its INSERT statements, then fails while planning the first SELECT. RESTCatalog reconstructs `TableSchema` from a REST `Schema` using `TableSchema.create`. The reconstructed creation time, schema-format version, and highest allocated field ID need not match the stored schema file. Comparing their full metadata therefore reports a replacement even when the table has not changed.
Bind REST file-table schemas to the physical schema at the same ID when loading raw catalog metadata, before caching it. Verify that the logical fields/keys/comment agree, preserve REST options, FileIO and catalog callbacks, and bind fallback branches independently. Cache hits do not rebind retained handles. The existing strict schema/snapshot generation checks remain unchanged, including rejection of a cached handle after same-path, same-schema-ID, same-field DROP/CREATE.
This adds a schema-file read on each REST table cache load, not on cache hits. Non-REST catalogs and non-file tables keep their existing path.
### Validation
- Four new tests reproduce the failure before the fix and pass afterward. They cover cache enabled/disabled, synthetic REST metadata, option/IO/environment preservation, same-shape replacement with a replacement pin, refresh, logical-schema mismatch, and fallback schemas.
- `mvn -f fe/pom.xml -pl :fe-connector-paimon -am install -Dmaven.build.cache.enabled=false` passed, including FE Checkstyle and plugin zip assembly: 1,460 tests discovered, zero failures/errors, one existing live-connectivity test skipped. The Paimon module has 806 passing tests and the same one skip.
- Extended the REST write regression with a refresh/re-read comparison after its existing golden assertion. Groovy syntax compilation and `git diff --check` passed.
Unit tests use real local Paimon schema files and SDK schema reconstruction with a recording REST catalog. A live REST/DLF environment was not exercised locally; the external regression remains for CI. The existing live test skips when `PAIMON_WAREHOUSE` is not configured.
### Release note
Fix false table-generation-change errors when reading tables through Paimon REST catalogs.
### Check List (For Author)
- [x] Unit Test
- [x] Regression test extension
- [x] Behavior changed: valid REST table schemas are accepted while retained generation checks remain strict
- [x] No documentation needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6206844749
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6210681672
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6208837971
<details>
<summary>ClickBench: <b>Total hot run time: 24.36 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 63a7b2ba1f63cfe4b2cd5da46820b22f82325726, data reload: false
query1 0.00 0.00 0.00
query2 0.10 0.05 0.05
query3 0.27 0.13 0.11
query4 1.61 0.12 0.12
query5 0.27 0.26 0.25
query6 1.16 0.68 0.68
query7 0.03 0.01 0.00
query8 0.06 0.04 0.04
query9 0.40 0.36 0.32
query10 0.57 0.56 0.60
query11 0.20 0.14 0.14
query12 0.19 0.15 0.14
query13 0.50 0.49 0.48
query14 1.02 1.03 1.00
query15 0.65 0.60 0.60
query16 0.33 0.36 0.31
query17 1.11 1.09 1.10
query18 0.23 0.22 0.21
query19 2.04 1.92 2.03
query20 0.02 0.01 0.01
query21 15.49 0.28 0.15
query22 4.49 0.06 0.05
query23 16.27 0.31 0.12
query24 3.34 0.57 0.40
query25 0.10 0.04 0.04
query26 1.02 0.25 0.15
query27 0.06 0.04 0.04
query28 1.81 0.62 0.40
query29 12.56 4.16 3.34
query30 0.30 0.14 0.14
query31 2.77 0.56 0.34
query32 3.22 0.60 0.51
query33 3.18 3.16 3.32
query34 15.54 3.96 3.38
query35 3.32 3.31 3.32
query36 0.55 0.45 0.48
query37 0.10 0.06 0.06
query38 0.05 0.04 0.03
query39 0.03 0.03 0.03
query40 0.17 0.14 0.14
query41 0.10 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.04 0.03
Total cold run time: 95.32 s
Total hot run time: 24.36 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #69158:
URL: https://github.com/apache/doris/pull/69158#discussion_r4290640419
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonSchemaPin.java:
##########
@@ -125,6 +126,29 @@ private static void validate(Table table, Map<String, String> pin, String path)
}
}
+ static FileStoreTable bindRestSchema(FileStoreTable table) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable) table;
+ return PaimonTableDecorators.rebuildFallbackPair(pair,
+ bindRestSchema(pair.wrapped()), bindRestSchema(pair.other()));
+ }
+ if (table instanceof DelegatedFileStoreTable) {
+ return PaimonTableDecorators.replaceWrapped(table,
+ bindRestSchema(((DelegatedFileStoreTable) table).wrapped()));
+ }
+ TableSchema retained = table.schema();
+ TableSchema persisted = table.schemaManager().schema(retained.id());
+ // REST reconstructs TableSchema from Schema, synthesizing timeMillis, version and highestFieldId.
+ // Bind physical identity only on catalog load: rebinding a cached handle could adopt a replacement.
+ if (!retained.copy(Collections.emptyMap()).toSchema()
+ .equals(persisted.copy(Collections.emptyMap()).toSchema())) {
+ throw new DorisConnectorException("Paimon REST schema does not match stored schema "
+ + retained.id() + "; retry the statement");
+ }
+ // Copy the schema on the existing table to preserve REST FileIO and catalog callbacks/options.
+ return table.copy(persisted.copy(retained.options()));
Review Comment:
Fixed in 63a7b2ba1f6. The initial REST handle now only locates schema files. We capture full physical identities, perform a fresh uncached REST lookup, and validate the physical identities again before binding and caching. The identity includes options and branch/path, and the wrapper shape and fallback leaves are checked as well. A replacement before the bracket uses fresh REST options; replacement during it rejects the load without caching it. UUID is not used as the fence.
Added deterministic tests for both interleavings, equal creation timestamps with changed bucket keys, fallback replacement/shape changes and changed location. A real local Paimon fixture writes 16 matching rows: mixed-generation options demonstrably lose rows, while the corrected load returns all 16. All nine REST schema tests pass. Full reactor install, Checkstyle and plugin assembly passed: 1,464 tests passed and one existing environment-dependent live test skipped. Live REST/DLF regression remains for CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6206847376
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6208781234
<details>
<summary>TPC-DS: <b>Total hot run time: 151808 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63a7b2ba1f63cfe4b2cd5da46820b22f82325726, data reload: false
query5 4309 595 453 453
query6 423 206 190 190
query7 4806 450 223 223
query8 315 168 155 155
query9 8757 3966 3958 3958
query10 462 308 256 256
query11 6063 3783 3524 3524
query12 146 102 86 86
query13 1254 431 307 307
query14 6160 4095 3741 3741
query14_1 3531 3521 3472 3472
query15 208 204 187 187
query16 947 401 461 401
query17 854 684 552 552
query18 2439 468 317 317
query19 214 166 137 137
query20 86 84 84 84
query21 207 134 114 114
query22 12940 12895 12878 12878
query23 13455 12821 12421 12421
query23_1 12655 12484 12481 12481
query24 7108 832 469 469
query24_1 488 468 479 468
query25 544 422 380 380
query26 1286 263 141 141
query27 2759 446 275 275
query28 4617 1959 1955 1955
query29 1604 647 430 430
query30 304 220 186 186
query31 1008 921 803 803
query32 142 85 91 85
query33 489 288 232 232
query34 907 862 482 482
query35 740 767 632 632
query36 787 826 714 714
query37 128 99 87 87
query38 1784 1742 1691 1691
query39 795 791 747 747
query39_1 731 723 755 723
query40 235 118 93 93
query41 67 63 62 62
query42 88 83 81 81
query43 358 389 324 324
query44 1276 686 691 686
query45 179 177 164 164
query46 849 951 566 566
query47 2936 2990 2908 2908
query48 297 312 214 214
query49 575 417 298 298
query50 664 272 219 219
query51 10626 10658 10456 10456
query52 74 78 67 67
query53 189 208 151 151
query54 251 210 175 175
query55 79 64 64 64
query56 227 209 220 209
query57 1596 1583 1568 1568
query58 287 296 256 256
query59 2134 2139 1955 1955
query60 290 239 223 223
query61 156 149 150 149
query62 404 350 286 286
query63 193 162 169 162
query64 2764 953 790 790
query65 3299 3234 3239 3234
query66 1781 407 304 304
query67 20167 20307 19780 19780
query68 2919 1001 599 599
query69 376 283 256 256
query70 857 788 783 783
query71 294 229 204 204
query72 2739 2578 2320 2320
query73 508 517 293 293
query74 4578 4466 4289 4289
query75 2348 2272 1940 1940
query76 2295 1003 595 595
query77 327 385 269 269
query78 9682 9651 9195 9195
query79 992 841 520 520
query80 1266 458 385 385
query81 572 314 269 269
query82 642 136 111 111
query83 310 203 197 197
query84 311 120 112 112
query85 928 523 379 379
query86 399 237 220 220
query87 2009 1959 1834 1834
query88 3601 2697 2675 2675
query89 340 290 260 260
query90 1919 174 179 174
query91 157 151 120 120
query92 96 88 88 88
query93 1049 1003 576 576
query94 639 272 290 272
query95 577 407 320 320
query96 660 522 232 232
query97 2490 2416 2340 2340
query98 163 155 148 148
query99 782 775 646 646
Total cold run time: 235121 ms
Total hot run time: 151808 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6208665686
<details>
<summary>TPC-H: <b>Total hot run time: 29446 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63a7b2ba1f63cfe4b2cd5da46820b22f82325726, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17589 5625 5522 5522
q2 2135 310 259 259
q3 10233 1401 855 855
q4 4671 474 346 346
q5 7436 791 527 527
q6 196 193 155 155
q7 778 813 597 597
q8 9247 1364 1113 1113
q9 5605 4232 4250 4232
q10 6822 1294 1015 1015
q11 427 251 234 234
q12 637 405 280 280
q13 18084 3698 2995 2995
q14 272 269 243 243
q15 q16 737 718 680 680
q17 1199 752 571 571
q18 7099 6245 6184 6184
q19 1273 957 598 598
q20 408 341 230 230
q21 5358 2519 2600 2519
q22 424 380 291 291
Total cold run time: 100630 ms
Total hot run time: 29446 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6626 6687 6695 6687
q2 787 587 552 552
q3 5460 5842 5206 5206
q4 2073 2154 1594 1594
q5 5826 5705 5880 5705
q6 247 189 148 148
q7 2258 1971 1803 1803
q8 3114 2696 2776 2696
q9 8057 7984 7550 7550
q10 3648 3590 3247 3247
q11 594 406 379 379
q12 657 686 508 508
q13 3197 3565 3020 3020
q14 296 282 261 261
q15 q16 661 679 607 607
q17 7796 7045 6893 6893
q18 13045 12266 12992 12266
q19 918 791 803 791
q20 2181 2154 1970 1970
q21 5795 4694 4810 4694
q22 507 446 402 402
Total cold run time: 73743 ms
Total hot run time: 66979 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6206850149
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6207496534
<details>
<summary>TPC-H: <b>Total hot run time: 29462 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8e0fd2d375a153f8107a59a24ffb6d3bef2e2152, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17609 5616 5549 5549
q2 2136 316 274 274
q3 10172 1402 822 822
q4 4677 463 346 346
q5 7435 790 529 529
q6 178 190 157 157
q7 782 791 604 604
q8 9253 1321 1091 1091
q9 5534 4264 4281 4264
q10 6827 1315 1017 1017
q11 434 257 240 240
q12 631 396 281 281
q13 18094 3574 2993 2993
q14 274 272 241 241
q15 q16 730 726 671 671
q17 1184 747 582 582
q18 7093 6236 6175 6175
q19 1103 957 592 592
q20 377 326 225 225
q21 5740 2523 2611 2523
q22 414 373 286 286
Total cold run time: 100677 ms
Total hot run time: 29462 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6468 6470 6479 6470
q2 727 580 554 554
q3 5491 5662 5244 5244
q4 2304 2119 1533 1533
q5 5913 5688 5590 5590
q6 259 193 145 145
q7 2245 1993 1815 1815
q8 3109 2782 2820 2782
q9 8063 7714 7543 7543
q10 3642 3579 3264 3264
q11 593 451 379 379
q12 661 683 495 495
q13 3255 3522 3016 3016
q14 287 289 268 268
q15 q16 665 674 605 605
q17 7861 6950 6908 6908
q18 13123 12205 13022 12205
q19 906 780 776 776
q20 2187 2194 1926 1926
q21 5738 4865 4807 4807
q22 500 439 398 398
Total cold run time: 73997 ms
Total hot run time: 66723 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6207656582
<details>
<summary>ClickBench: <b>Total hot run time: 24.46 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8e0fd2d375a153f8107a59a24ffb6d3bef2e2152, data reload: false
query1 0.00 0.00 0.00
query2 0.09 0.05 0.05
query3 0.27 0.12 0.11
query4 1.61 0.13 0.12
query5 0.25 0.25 0.25
query6 1.17 0.69 0.68
query7 0.04 0.01 0.01
query8 0.06 0.04 0.04
query9 0.41 0.35 0.35
query10 0.57 0.60 0.61
query11 0.20 0.14 0.15
query12 0.19 0.15 0.15
query13 0.49 0.50 0.50
query14 1.03 1.02 0.99
query15 0.64 0.60 0.61
query16 0.32 0.35 0.32
query17 1.12 1.14 1.19
query18 0.23 0.21 0.21
query19 2.03 1.91 2.05
query20 0.02 0.01 0.01
query21 15.51 0.27 0.14
query22 4.35 0.05 0.06
query23 16.23 0.32 0.12
query24 3.33 0.79 0.39
query25 0.10 0.06 0.04
query26 1.22 0.25 0.15
query27 0.05 0.04 0.03
query28 1.58 0.63 0.39
query29 12.49 4.14 3.33
query30 0.28 0.14 0.14
query31 2.80 0.57 0.33
query32 3.22 0.60 0.53
query33 3.20 3.20 3.16
query34 15.42 3.95 3.38
query35 3.32 3.31 3.35
query36 0.55 0.43 0.45
query37 0.09 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.16 0.15 0.15
query41 0.08 0.02 0.02
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 94.89 s
Total hot run time: 24.46 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6210641229
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/23)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69158_63a7b2ba1f63cfe4b2cd5da46820b22f82325726_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69158_63a7b2ba1f63cfe4b2cd5da46820b22f82325726_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6210680999
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #69158:
URL: https://github.com/apache/doris/pull/69158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6207602187
<details>
<summary>TPC-DS: <b>Total hot run time: 151110 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8e0fd2d375a153f8107a59a24ffb6d3bef2e2152, data reload: false
query5 4368 588 458 458
query6 428 210 194 194
query7 4816 451 229 229
query8 326 188 152 152
query9 8752 3914 3916 3914
query10 488 312 249 249
query11 6067 3804 3519 3519
query12 150 95 86 86
query13 1258 444 315 315
query14 6112 4003 3727 3727
query14_1 3511 3468 3487 3468
query15 203 201 180 180
query16 947 425 409 409
query17 857 665 543 543
query18 2428 427 310 310
query19 197 164 143 143
query20 86 86 92 86
query21 210 137 111 111
query22 12936 13066 12832 12832
query23 13515 12843 12376 12376
query23_1 12591 12475 12529 12475
query24 7092 813 454 454
query24_1 466 471 480 471
query25 510 389 331 331
query26 1254 253 127 127
query27 2774 436 260 260
query28 4646 1968 1953 1953
query29 1541 571 436 436
query30 307 217 182 182
query31 1005 922 799 799
query32 144 88 89 88
query33 498 277 227 227
query34 898 845 485 485
query35 724 744 632 632
query36 825 805 711 711
query37 128 102 85 85
query38 1807 1727 1693 1693
query39 780 818 749 749
query39_1 733 740 750 740
query40 218 111 92 92
query41 65 61 62 61
query42 93 85 81 81
query43 360 387 324 324
query44 1267 675 680 675
query45 178 173 158 158
query46 848 963 550 550
query47 2736 3030 2913 2913
query48 299 300 232 232
query49 573 424 290 290
query50 667 293 206 206
query51 10355 10600 10683 10600
query52 78 77 69 69
query53 181 206 152 152
query54 236 199 197 197
query55 73 67 65 65
query56 223 210 231 210
query57 1592 1580 1560 1560
query58 282 259 240 240
query59 2118 2221 1921 1921
query60 304 241 227 227
query61 139 146 145 145
query62 386 342 283 283
query63 191 168 169 168
query64 2734 934 769 769
query65 3297 3228 3227 3227
query66 1768 453 296 296
query67 20254 20181 19871 19871
query68 2956 984 612 612
query69 405 290 247 247
query70 848 768 761 761
query71 299 229 203 203
query72 2736 2609 1876 1876
query73 514 522 287 287
query74 4573 4477 4279 4279
query75 2340 2277 1937 1937
query76 2314 1012 615 615
query77 332 380 265 265
query78 9632 9661 9130 9130
query79 889 834 505 505
query80 489 404 363 363
query81 517 321 277 277
query82 208 135 113 113
query83 188 195 188 188
query84 293 125 108 108
query85 759 414 372 372
query86 256 234 229 229
query87 1972 1962 1831 1831
query88 3539 2659 2651 2651
query89 331 295 260 260
query90 2092 175 173 173
query91 159 161 124 124
query92 96 88 88 88
query93 982 991 568 568
query94 450 318 290 290
query95 587 396 322 322
query96 667 524 227 227
query97 2418 2402 2313 2313
query98 156 166 151 151
query99 763 765 653 653
Total cold run time: 232340 ms
Total hot run time: 151110 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69158:
URL: https://github.com/apache/doris/pull/69158#discussion_r4290406487
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonSchemaPin.java:
##########
@@ -125,6 +126,29 @@ private static void validate(Table table, Map<String, String> pin, String path)
}
}
+ static FileStoreTable bindRestSchema(FileStoreTable table) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable) table;
+ return PaimonTableDecorators.rebuildFallbackPair(pair,
+ bindRestSchema(pair.wrapped()), bindRestSchema(pair.other()));
+ }
+ if (table instanceof DelegatedFileStoreTable) {
+ return PaimonTableDecorators.replaceWrapped(table,
+ bindRestSchema(((DelegatedFileStoreTable) table).wrapped()));
+ }
+ TableSchema retained = table.schema();
+ TableSchema persisted = table.schemaManager().schema(retained.id());
+ // REST reconstructs TableSchema from Schema, synthesizing timeMillis, version and highestFieldId.
+ // Bind physical identity only on catalog load: rebinding a cached handle could adopt a replacement.
+ if (!retained.copy(Collections.emptyMap()).toSchema()
+ .equals(persisted.copy(Collections.emptyMap()).toSchema())) {
+ throw new DorisConnectorException("Paimon REST schema does not match stored schema "
+ + retained.id() + "; retry the statement");
+ }
+ // Copy the schema on the existing table to preserve REST FileIO and catalog callbacks/options.
+ return table.copy(persisted.copy(retained.options()));
Review Comment:
[P1] Keep REST metadata and physical schema in the same table generation. If a table is dropped/recreated at the same path after super.getTable() returns but before schema(retained.id()) reads the file, a replacement with the same schema ID/fields/keys/comment passes this comparison. This copy then combines the replacement's creation metadata with the old table's options. For example, primary keys (x,y) with bucket=2 can change bucket-key from x to y on recreation: later schema/snapshot pins all describe the replacement and pass, but WHERE x=k still prunes buckets using the old bucket-key=x, dropping matching replacement rows. REST snapshot reads are keyed by identifier, so retaining the old UUID does not fence this read. Establish that both reads refer to one generation (or reject/retry the load) before installing the physical identity, and cover replacement during binding; the current replacement test swaps files only after binding has completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6207434609
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Bind REST schemas to stored table generations [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69158:
URL: https://github.com/apache/doris/pull/69158#issuecomment-6207434912
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org