You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/10/10 19:01:48 UTC
[PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
morningman opened a new pull request, #68897:
URL: https://github.com/apache/doris/pull/68897
### What problem does this PR solve?
Issue Number: None
Related PR: #68885
Problem Summary:
**In short.** Master cannot read a Paimon 1.4 chain table: every data scan fails in planning with Paimon's internal `This is a bug, perhaps there is a recursive call.` This PR keeps the chain group intact wherever the Paimon connector and the BE JNI scanner rebuild a fallback pair, and makes a data scan of a chain table fail with a clear "not supported yet" message, because the rest of the scan path does not plan chain partitions and chain splits yet. The review of #68885 (closed since) found the BE half of the problem.
**Background.** Paimon 1.4 builds a chain table (`chain-table.enabled` with `scan.fallback-snapshot-branch` and `scan.fallback-delta-branch`) as `FallbackReadFileStoreTable(main, ChainGroupReadTable(snapshot, delta), true)`. `ChainGroupReadTable` is itself a fallback pair. It plans and reads the chain: a partition missing from the snapshot branch is read as the latest earlier snapshot partition merged with the delta branch. Paimon's `FallbackReadFileStoreTable` constructor rejects any other pair nested inside a pair. The Paimon connector and the BE JNI scanner rebuild fallback pairs branch by branch in seven places:
- the manifest-parallelism bound on the BE (`PaimonJniScanner`, taken when the FE cap is below the BE's core count) and on the FE (`PaimonReaderOptions`);
- the catalog-loader strip (`dropCatalogLoader`) and the catalog snapshot pin (`pinCatalogSnapshot`) of the table sent to the BE;
- the restore of a statement pin: pinned fallback snapshots (`copyWithPinnedFallback`) and bound schemas (`restoreBoundSchema`).
**The problem, and what it cost.** Each of these recursed into the chain group and rebuilt it either as a plain pair, which the outer constructor rejects, or as a plain table of the snapshot branch, which silently drops the delta branch. On a local FE + BE with a file-system Paimon catalog and a chain table created and written through the Paimon API:
| Read | Paimon's own read | master |
|---|---|---|
| `SELECT * FROM chain_t` | 5 rows: 2 from the snapshot branch, 3 chained from the snapshot and delta branches | fails: `This is a bug, perhaps there is a recursive call.` (first from `PaimonScanParams.restoreBoundSchema`) |
| a `$files` scan when the FE cap is below the BE's core count | | the BE manifest bound fails the same way (unit test; a single host does not take that path) |
Keeping the chain group is not enough to read a chain table. Doris plans neither the partitions a chain table reads from its branches nor the chain splits. With only the rebuilds fixed, a chain table whose main branch is empty reads as empty (0 partitions, 0 splits), and one with main data fails with `Fail to parse columnsFromPath, expected: [dt], filePath: connector://virtual`. branch-4.x has the same manifest-bound rebuild and no chain support either.
**How this PR fixes it.**
- `PaimonTableDecorators.rebuildFallbackPair` rebuilds a `ChainGroupReadTable` as a group and any other pair as before. The FE manifest bound, `copyWithPinnedFallback` and `restoreBoundSchema` use it, and `dropCatalogLoader` and `pinCatalogSnapshot` now recurse into the nested pair. The BE manifest bound keeps the group the same way.
- `PaimonScanPlanProvider.resolveScanTable` rejects a data scan of a chain table (a fallback pair over a `ChainGroupReadTable`) with `Reading Paimon chain table db.tbl (chain-table.enabled) is not supported yet`. System tables and a branch read on its own (`@branch(...)`) are plain tables and are not affected.
**Results.** Same cluster and table:
| Read | before | after |
|---|---|---|
| `SELECT ... FROM chain_t` | `This is a bug, perhaps there is a recursive call.` | `Reading Paimon chain table d.chain_t (chain-table.enabled) is not supported yet` |
| ``chain_t$files`` | reads the main branch's files | same |
| `chain_t@branch(snapshot)`, `chain_t@branch(delta)` | read that branch's rows | same |
| an ordinary Paimon table | reads | same |
### Release note
Reading a Paimon chain table (`chain-table.enabled`) fails with `Reading Paimon chain table ... is not supported yet` instead of an internal Paimon error.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `PaimonJniScannerTest.testBackendManifestCapKeepsTheChainGroupOfAChainTable`, `PaimonReaderOptionsTest.testRuntimeSafeTableKeepsTheChainGroupOfAChainTable`: the BE and FE manifest bounds keep the chain group. The BE case fails on master with the recursive-call error.
- `PaimonBackendBoundTableTest.chainTableKeepsItsChainGroupWithoutTheCatalogLoader`: on master the chain group comes back as a plain `PrimaryKeyFileStoreTable`. `PaimonBackendBoundTableTest.scanOfAChainTableFailsClearly`: a data scan of a chain table fails, and a branch read on its own resolves.
- `PaimonStatementSchemaTest.statementPinKeepsTheChainOfAChainTable`: a real chain table, pinned and restored, keeps the group and reads the same rows as Paimon's own read at that snapshot.
- Rebuilding the group as a plain pair, or dropping the scan check, makes exactly the matching new cases fail. The Paimon connector (699) and paimon-scanner module (114) tests pass.
- [x] Manual test (add detailed scripts or steps below)
- The tables above. The local BE predates #68785, so the FE under test is this change on master `038e2274ef6`; the connector code it touches is the same there.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. A data scan of a Paimon chain table fails with a clear message instead of a Paimon internal error. Nothing that read before changes.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68897:
URL: https://github.com/apache/doris/pull/68897#discussion_r4238874498
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1553,15 +1562,17 @@ static FileStoreTable dropCatalogLoader(FileStoreTable dataTable) {
if (dataTable.catalogEnvironment().catalogLoader() == null) {
return dataTable;
}
- FileStoreTable undecorated = PaimonTableDecorators.unwrapToFallbackOrBase(dataTable);
- if (undecorated instanceof FallbackReadFileStoreTable) {
- FallbackReadFileStoreTable fallbackReadTable = (FallbackReadFileStoreTable) undecorated;
- return new FallbackReadFileStoreTable(
- rebuildWithoutCatalogLoader(fallbackReadTable.wrapped()),
- rebuildWithoutCatalogLoader(fallbackReadTable.other()),
- PaimonReaderOptions.isWrappedFirst(fallbackReadTable));
+ return dropCatalogLoaderOfBranches(PaimonTableDecorators.unwrapToFallbackOrBase(dataTable));
+ }
+
+ private static FileStoreTable dropCatalogLoaderOfBranches(FileStoreTable table) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ // A chain table nests its snapshot and delta branches in a pair of their own; rebuild those too.
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable) table;
+ return PaimonTableDecorators.rebuildFallbackPair(pair,
Review Comment:
[P2] Keep this system-source payload consumable by the previous BE. It now serializes a preserved `ChainGroupReadTable` for chain-backed system views. During a rolling upgrade, a prior-head BE with fewer CPUs can lower a child `scan.manifest.parallelism` value, rebuild that group as a plain `FallbackReadFileStoreTable` (its pre-change scanner behavior), and then fail when constructing the outer pair because Paimon requires a nested `ChainGroupReadTable`. For example, an FE cap of 16, BE cap of 8, and snapshot-branch value of 16 trigger this on a supported `$files` scan with a main-branch marker. Gate affected system scans or send a payload the previous BE can consume. See [FallbackReadFileStoreTable](https://github.com/apache/paimon/blob/release-1.4.2/paimon-core/src/main/java/org/apache/paimon/table/FallbackReadFileStoreTable.java).
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -346,6 +346,12 @@ Table resolveTable(PaimonTableHandle paimonHandle) {
*/
Table resolveScanTable(PaimonTableHandle paimonHandle) {
Table table = resolveTable(paimonHandle);
+ if (!paimonHandle.isSystemTable() && PaimonTableDecorators.isChainTable(table)) {
Review Comment:
[P1] Reject chain-backed `$ro` reads at this boundary. The new `isSystemTable()` exception leaves `t$ro` open even though it is a data scan over the same outer chain pair. In Paimon 1.4.2, `ReadOptimizedTable.newScan()` passes a callback through the outer fallback; for the nested `ChainGroupReadTable`, that callback reads its delegated snapshot branch rather than invoking the chain planner. A partition present only in delta, even with files eligible for the read-optimized scan, can therefore disappear from `SELECT ... FROM t$ro`; an emitted fallback `DataSplit` also does not meet the group reader's `ChainSplit` requirement. The SDK behavior predates this guard, but the new selective refusal leaves this unsupported read exposed. Reject chain-backed `$ro` at handle construction or scanning until it has a chain-aware path, and cover a delta-only partition. See [ReadOptimizedTable](https://github.com/apache/paimon/blob/release-1.4.2/paimon-core/src/main/java/org/apache/paimon/table/sy
stem/ReadOptimizedTable.java) and [ChainGroupReadTable](https://github.com/apache/paimon/blob/release-1.4.2/paimon-core/src/main/java/org/apache/paimon/table/ChainGroupReadTable.java).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6140186080
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6101253568
<details>
<summary>TPC-H: <b>Total hot run time: 29264 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f64111fb2aa6691ca47ad41e65100112560bfa9f, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17581 5634 5550 5550
q2 2096 310 254 254
q3 10236 1391 844 844
q4 4687 475 346 346
q5 7429 794 541 541
q6 186 185 152 152
q7 778 803 610 610
q8 9246 1365 1141 1141
q9 5727 4475 4444 4444
q10 6847 1299 1030 1030
q11 427 254 246 246
q12 624 397 282 282
q13 18085 3082 2399 2399
q14 273 263 247 247
q15 q16 736 727 662 662
q17 1179 745 588 588
q18 7097 6272 6204 6204
q19 1113 956 603 603
q20 388 339 227 227
q21 5304 2591 2691 2591
q22 410 335 303 303
Total cold run time: 100449 ms
Total hot run time: 29264 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6693 6656 6836 6656
q2 743 587 545 545
q3 5527 5699 5266 5266
q4 2063 2157 1567 1567
q5 5856 5665 5797 5665
q6 242 194 143 143
q7 2260 2034 1807 1807
q8 3141 2752 2681 2681
q9 8233 8197 7931 7931
q10 3651 3596 3283 3283
q11 596 417 372 372
q12 660 693 495 495
q13 2764 3058 2375 2375
q14 283 289 268 268
q15 q16 661 673 599 599
q17 7751 7103 6914 6914
q18 13127 12231 13142 12231
q19 908 786 772 772
q20 2192 2174 1960 1960
q21 5791 4756 4763 4756
q22 486 451 410 410
Total cold run time: 73628 ms
Total hot run time: 66696 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #68897:
URL: https://github.com/apache/doris/pull/68897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6105332652
<details>
<summary>TPC-DS: <b>Total hot run time: 151999 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26eebd6752bad84f1804b1de130494899699c9cc, data reload: false
query5 4314 575 458 458
query6 422 208 185 185
query7 4798 457 226 226
query8 326 182 169 169
query9 8734 4030 4004 4004
query10 473 307 247 247
query11 5982 3415 3194 3194
query12 147 87 83 83
query13 1273 426 308 308
query14 6553 4835 4538 4538
query14_1 4294 4294 4272 4272
query15 209 200 185 185
query16 972 469 448 448
query17 916 658 546 546
query18 2420 423 309 309
query19 192 163 136 136
query20 84 80 83 80
query21 212 158 107 107
query22 13039 13033 12889 12889
query23 13171 12442 12062 12062
query23_1 12197 12190 12141 12141
query24 7098 821 454 454
query24_1 464 457 458 457
query25 496 389 333 333
query26 1257 259 132 132
query27 2766 438 267 267
query28 4620 1918 1921 1918
query29 1529 580 436 436
query30 292 227 184 184
query31 903 763 659 659
query32 135 90 88 88
query33 522 297 241 241
query34 900 860 501 501
query35 761 765 642 642
query36 810 796 737 737
query37 131 114 85 85
query38 1780 1759 1736 1736
query39 715 717 676 676
query39_1 694 659 685 659
query40 216 115 93 93
query41 66 64 67 64
query42 84 82 81 81
query43 361 384 338 338
query44 1320 680 690 680
query45 176 183 159 159
query46 827 955 572 572
query47 3020 2991 2938 2938
query48 297 298 210 210
query49 568 388 293 293
query50 690 275 210 210
query51 10386 10402 10322 10322
query52 78 92 73 73
query53 183 209 152 152
query54 234 203 188 188
query55 81 73 66 66
query56 240 238 228 228
query57 1623 1594 1581 1581
query58 300 268 248 248
query59 2363 2377 2138 2138
query60 296 256 231 231
query61 169 168 168 168
query62 407 345 297 297
query63 210 175 183 175
query64 2899 1069 886 886
query65 3456 3396 3376 3376
query66 1797 489 299 299
query67 20196 20201 19856 19856
query68 3310 981 598 598
query69 391 293 251 251
query70 916 861 858 858
query71 305 227 211 211
query72 2808 2623 2295 2295
query73 546 529 295 295
query74 4622 4459 4312 4312
query75 2350 2283 1938 1938
query76 2376 1016 615 615
query77 372 383 306 306
query78 9054 9016 8546 8546
query79 971 853 512 512
query80 1202 410 347 347
query81 591 322 277 277
query82 590 133 107 107
query83 300 197 182 182
query84 309 119 96 96
query85 858 413 365 365
query86 399 236 231 231
query87 2014 1943 1818 1818
query88 3638 2743 2669 2669
query89 326 294 262 262
query90 1855 187 184 184
query91 161 148 122 122
query92 96 82 88 82
query93 1084 995 576 576
query94 654 306 274 274
query95 574 351 317 317
query96 671 521 232 232
query97 2432 2420 2294 2294
query98 151 149 142 142
query99 780 758 654 654
Total cold run time: 235597 ms
Total hot run time: 151999 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68897:
URL: https://github.com/apache/doris/pull/68897#discussion_r4240173755
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -778,6 +779,11 @@ private static Table applyManifestParallelismBound(
}
// Each branch owns an independent planner setting; a smaller sibling is not an
// execution ceiling and must never throttle the other branch.
+ if (pair instanceof ChainGroupReadTable) {
+ // A chain table nests this group over its snapshot and delta branches. The group plans
Review Comment:
[P2] Keep chain fallback splits out of `$files`'s `DataSplit` reader. FE can emit a `$files` marker for main partition P, then main can drop P before BE reads that marker. With this preserved group in the system-source payload, BE replanning falls back to `FallbackSplitImpl(ChainSplit)` for snapshot-branch P; Paimon [FilesRead](https://github.com/apache/paimon/blob/release-1.4.2/paimon-core/src/main/java/org/apache/paimon/table/system/FilesTable.java) casts every replanned split to `DataSplit`, so the query fails with `ClassCastException`. The previous flattened snapshot fallback returned `FallbackDataSplit`, which passed that cast. Align marker and reader snapshots or prevent this wrapper from reaching `$files` until its reader can handle the chain split, and test the drop-after-marker case.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1553,15 +1562,17 @@ static FileStoreTable dropCatalogLoader(FileStoreTable dataTable) {
if (dataTable.catalogEnvironment().catalogLoader() == null) {
return dataTable;
}
- FileStoreTable undecorated = PaimonTableDecorators.unwrapToFallbackOrBase(dataTable);
- if (undecorated instanceof FallbackReadFileStoreTable) {
- FallbackReadFileStoreTable fallbackReadTable = (FallbackReadFileStoreTable) undecorated;
- return new FallbackReadFileStoreTable(
- rebuildWithoutCatalogLoader(fallbackReadTable.wrapped()),
- rebuildWithoutCatalogLoader(fallbackReadTable.other()),
- PaimonReaderOptions.isWrappedFirst(fallbackReadTable));
+ return dropCatalogLoaderOfBranches(PaimonTableDecorators.unwrapToFallbackOrBase(dataTable));
+ }
+
+ private static FileStoreTable dropCatalogLoaderOfBranches(FileStoreTable table) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ // A chain table nests its snapshot and delta branches in a pair of their own; rebuild those too.
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable) table;
+ return PaimonTableDecorators.rebuildFallbackPair(pair,
+ dropCatalogLoaderOfBranches(pair.wrapped()), dropCatalogLoaderOfBranches(pair.other()));
}
Review Comment:
[P2] Pin the chain system source before stripping its catalog loaders. For a plain chain-backed `$partitions` read in a version-managed catalog, this recursion retains the delta leaf in `doris.serialized-system-source`, but that source never receives the `pinCatalogSnapshot` applied to `serializedTable`. The BE always rebuilds from the side-channel source, so if delta snapshot 2 exists on storage while its catalog pointer remains at 1, Paimon's partition listing can return the unpublished delta-only partition. Carry catalog-visible coordinates for each branch into the source the BE actually uses, then test a delta-only partition in that publication window. See Paimon [PartitionsRead](https://github.com/apache/paimon/blob/release-1.4.2/paimon-core/src/main/java/org/apache/paimon/table/system/PartitionsTable.java) and [ChainGroupReadTable](https://github.com/apache/paimon/blob/release-1.4.2/paimon-core/src/main/java/org/apache/paimon/table/ChainGroupReadTable.java).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6101091124
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6105365769
<details>
<summary>ClickBench: <b>Total hot run time: 24.66 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26eebd6752bad84f1804b1de130494899699c9cc, data reload: false
query1 0.01 0.01 0.00
query2 0.09 0.06 0.04
query3 0.27 0.12 0.12
query4 1.61 0.14 0.13
query5 0.28 0.26 0.25
query6 1.16 0.68 0.68
query7 0.04 0.01 0.01
query8 0.06 0.03 0.04
query9 0.44 0.36 0.37
query10 0.63 0.65 0.64
query11 0.21 0.14 0.14
query12 0.18 0.15 0.14
query13 0.50 0.50 0.50
query14 0.97 0.96 0.97
query15 0.64 0.61 0.61
query16 0.36 0.35 0.37
query17 1.16 1.11 1.13
query18 0.26 0.23 0.23
query19 2.10 2.04 1.96
query20 0.02 0.02 0.01
query21 15.49 0.26 0.15
query22 4.45 0.06 0.05
query23 16.25 0.33 0.12
query24 2.97 0.70 0.44
query25 0.08 0.04 0.06
query26 1.82 0.24 0.16
query27 0.06 0.03 0.05
query28 2.40 0.65 0.41
query29 12.47 4.10 3.28
query30 0.28 0.15 0.14
query31 2.77 0.56 0.34
query32 3.22 0.61 0.52
query33 3.17 3.20 3.20
query34 15.48 4.07 3.40
query35 3.35 3.35 3.35
query36 0.57 0.45 0.43
query37 0.09 0.06 0.06
query38 0.05 0.03 0.04
query39 0.04 0.02 0.02
query40 0.17 0.15 0.13
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 96.34 s
Total hot run time: 24.66 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6105262290
<details>
<summary>TPC-H: <b>Total hot run time: 29069 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26eebd6752bad84f1804b1de130494899699c9cc, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17616 5671 5557 5557
q2 2120 297 245 245
q3 10241 1388 841 841
q4 4675 476 346 346
q5 7454 798 530 530
q6 188 186 158 158
q7 763 808 628 628
q8 9258 1375 1164 1164
q9 5721 4460 4444 4444
q10 6819 1294 1014 1014
q11 442 258 237 237
q12 630 396 281 281
q13 18070 3054 2360 2360
q14 281 263 244 244
q15 q16 735 732 649 649
q17 1161 750 568 568
q18 7196 6207 6158 6158
q19 1268 952 616 616
q20 395 339 223 223
q21 6051 2509 2631 2509
q22 417 358 297 297
Total cold run time: 101501 ms
Total hot run time: 29069 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6599 6609 6832 6609
q2 710 569 561 561
q3 5611 5730 5226 5226
q4 2072 2121 1595 1595
q5 5677 5683 5771 5683
q6 291 186 146 146
q7 2182 2026 1800 1800
q8 3081 2816 2786 2786
q9 8326 8130 8097 8097
q10 3648 3590 3248 3248
q11 598 397 369 369
q12 659 687 504 504
q13 2738 3066 2375 2375
q14 286 281 255 255
q15 q16 664 695 604 604
q17 7752 6948 6838 6838
q18 13038 12225 13060 12225
q19 892 779 791 779
q20 2170 2166 1925 1925
q21 5788 4706 4812 4706
q22 491 454 435 435
Total cold run time: 73273 ms
Total hot run time: 66766 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6102814011
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68897_f64111fb2aa6691ca47ad41e65100112560bfa9f/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68897_f64111fb2aa6691ca47ad41e65100112560bfa9f/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6105098273
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6140186040
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6101344690
<details>
<summary>TPC-DS: <b>Total hot run time: 151268 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f64111fb2aa6691ca47ad41e65100112560bfa9f, data reload: false
query5 4302 574 447 447
query6 415 207 191 191
query7 4805 440 222 222
query8 315 179 163 163
query9 8766 3940 4005 3940
query10 474 319 263 263
query11 6016 3433 3187 3187
query12 145 87 81 81
query13 1251 428 303 303
query14 6537 4783 4490 4490
query14_1 4254 4231 4262 4231
query15 218 199 180 180
query16 940 439 374 374
query17 841 643 522 522
query18 2413 416 296 296
query19 186 151 129 129
query20 78 82 76 76
query21 210 129 109 109
query22 13060 13035 12720 12720
query23 13025 12396 12086 12086
query23_1 12170 12247 12144 12144
query24 7100 828 465 465
query24_1 473 467 468 467
query25 536 419 362 362
query26 1275 255 138 138
query27 2772 434 268 268
query28 4659 1917 1920 1917
query29 1579 601 449 449
query30 294 222 184 184
query31 886 769 647 647
query32 146 89 90 89
query33 514 298 244 244
query34 1012 876 495 495
query35 753 788 667 667
query36 853 820 777 777
query37 138 104 88 88
query38 1796 1742 1715 1715
query39 722 729 701 701
query39_1 662 635 664 635
query40 220 136 103 103
query41 74 71 69 69
query42 88 82 79 79
query43 359 378 315 315
query44 1285 698 696 696
query45 184 179 166 166
query46 843 970 575 575
query47 2982 2718 2782 2718
query48 304 310 217 217
query49 580 411 299 299
query50 693 337 208 208
query51 10455 10360 10310 10310
query52 75 79 70 70
query53 183 217 152 152
query54 237 188 185 185
query55 73 70 61 61
query56 219 208 197 197
query57 1599 1601 1545 1545
query58 272 260 244 244
query59 2361 2367 2135 2135
query60 276 230 218 218
query61 142 140 146 140
query62 388 336 284 284
query63 195 157 162 157
query64 2763 929 759 759
query65 3407 3358 3380 3358
query66 1828 417 297 297
query67 20410 20045 20091 20045
query68 3222 985 588 588
query69 398 269 249 249
query70 931 828 874 828
query71 290 225 196 196
query72 2878 2606 2289 2289
query73 525 520 294 294
query74 4607 4474 4308 4308
query75 2348 2304 1945 1945
query76 2375 1018 601 601
query77 355 383 300 300
query78 8942 8966 8479 8479
query79 866 892 522 522
query80 490 436 339 339
query81 527 327 276 276
query82 210 134 118 118
query83 195 197 174 174
query84 285 124 100 100
query85 785 409 366 366
query86 252 241 234 234
query87 2009 1994 1861 1861
query88 3626 2678 2668 2668
query89 329 296 256 256
query90 2120 179 184 179
query91 163 147 118 118
query92 86 88 87 87
query93 974 984 561 561
query94 452 300 274 274
query95 588 359 318 318
query96 651 516 227 227
query97 2447 2452 2332 2332
query98 159 148 152 148
query99 776 766 657 657
Total cold run time: 233677 ms
Total hot run time: 151268 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6105098134
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6106387060
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68897_26eebd6752bad84f1804b1de130494899699c9cc_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68897_26eebd6752bad84f1804b1de130494899699c9cc_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6140185380
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6102773374
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68897_f64111fb2aa6691ca47ad41e65100112560bfa9f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68897_f64111fb2aa6691ca47ad41e65100112560bfa9f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6101091395
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68897:
URL: https://github.com/apache/doris/pull/68897#discussion_r4240131211
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -346,6 +346,12 @@ Table resolveTable(PaimonTableHandle paimonHandle) {
*/
Table resolveScanTable(PaimonTableHandle paimonHandle) {
Table table = resolveTable(paimonHandle);
+ if (!paimonHandle.isSystemTable() && PaimonTableDecorators.isChainTable(table)) {
Review Comment:
Not changing this: `t$ro` on a chain table returns what Paimon itself returns, on master and with this PR.
`$ro` is Paimon's view of the latest full compaction. `ReadOptimizedTable.newScan()` hands its callback to the outer fallback pair, and for the nested group that callback plans with the snapshot branch's reader, so a partition held only by the delta branch is not in Paimon's own `t$ro` either; the delta branch's compacted files are increments, not that partition. The check at this line could not catch `$ro` in any case: the handle's table is a `ReadOptimizedTable`, not a `FileStoreTable`, so `isChainTable` is false for it.
Checked with Paimon 1.4.2 in jshell on a chain table with main 20231231, snapshot 20240101 and a delta-only 20240102, every branch fully compacted. Paimon's own `t$ro` returns `[20231231 9 z, 20240101 1 a, 20240101 2 b]`, and reading the same splits through master's backend payload and through this PR's returns the same three rows. With this PR Paimon logs the same "Reading from supplemental branch has problems" error per fallback split that its own table logs.
Refusing `$ro` would fail a read that works on master. A chain-aware `$ro`, if Paimon defines one, belongs with the chain read support this PR leaves out.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1553,15 +1562,17 @@ static FileStoreTable dropCatalogLoader(FileStoreTable dataTable) {
if (dataTable.catalogEnvironment().catalogLoader() == null) {
return dataTable;
}
- FileStoreTable undecorated = PaimonTableDecorators.unwrapToFallbackOrBase(dataTable);
- if (undecorated instanceof FallbackReadFileStoreTable) {
- FallbackReadFileStoreTable fallbackReadTable = (FallbackReadFileStoreTable) undecorated;
- return new FallbackReadFileStoreTable(
- rebuildWithoutCatalogLoader(fallbackReadTable.wrapped()),
- rebuildWithoutCatalogLoader(fallbackReadTable.other()),
- PaimonReaderOptions.isWrappedFirst(fallbackReadTable));
+ return dropCatalogLoaderOfBranches(PaimonTableDecorators.unwrapToFallbackOrBase(dataTable));
+ }
+
+ private static FileStoreTable dropCatalogLoaderOfBranches(FileStoreTable table) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ // A chain table nests its snapshot and delta branches in a pair of their own; rebuild those too.
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable) table;
+ return PaimonTableDecorators.rebuildFallbackPair(pair,
Review Comment:
Confirmed, and not changed, for three reasons:
- It only happens when the FE is upgraded before the BE. Doris upgrades BEs first, and this PR's BE reads both the new payload and the previous FE's, in which the group has been replaced by a plain table at the snapshot branch (`testBackendCapStillReadsTheChainTablePayloadOfAnOlderFrontend`, 26eebd6752b).
- It fails the scan at scanner initialization rather than returning rows, and stops once that BE is upgraded.
- The payload the previous BE can consume is that flattened one, and it loses the delta branch. Paimon resolves `$partitions` on the backend: for a chain table whose delta branch alone holds 20240102, the flattened payload lists `[20231231, 20240101]`, while the payload with the group lists Paimon's own `[20231231, 20240101, 20240102]` (Paimon 1.4.2 in jshell). That would trade a temporary error for a silently wrong result.
The trigger is wider than the example: calling the pre-PR scanner's cap path with an FE cap of 16 fails whenever the BE has more cores than the cap, since the absent option is then filled in and the nested pair rebuilt, not only when a branch sets a larger value. 26eebd6752b also covers the system-source path (`testBackendCapKeepsTheChainGroupOfAChainTableSystemSource`), and the PR description now states the BE-first upgrade order and that `$partitions` lists the delta-only partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6106363329
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68897_26eebd6752bad84f1804b1de130494899699c9cc/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68897_26eebd6752bad84f1804b1de130494899699c9cc/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6101394199
<details>
<summary>ClickBench: <b>Total hot run time: 24.27 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f64111fb2aa6691ca47ad41e65100112560bfa9f, data reload: false
query1 0.00 0.00 0.01
query2 0.10 0.05 0.05
query3 0.26 0.12 0.11
query4 1.61 0.12 0.12
query5 0.27 0.25 0.24
query6 1.17 0.68 0.67
query7 0.04 0.01 0.00
query8 0.05 0.03 0.04
query9 0.42 0.35 0.36
query10 0.57 0.55 0.58
query11 0.21 0.14 0.14
query12 0.18 0.15 0.15
query13 0.49 0.49 0.51
query14 0.99 0.97 0.97
query15 0.63 0.60 0.62
query16 0.35 0.34 0.36
query17 1.19 1.14 1.09
query18 0.22 0.20 0.21
query19 2.18 1.99 1.89
query20 0.02 0.01 0.02
query21 15.49 0.27 0.14
query22 4.50 0.05 0.05
query23 16.25 0.31 0.12
query24 2.97 0.72 0.40
query25 0.12 0.05 0.04
query26 1.37 0.25 0.14
query27 0.05 0.05 0.03
query28 1.55 0.61 0.39
query29 12.53 4.13 3.25
query30 0.28 0.14 0.14
query31 2.77 0.59 0.34
query32 3.22 0.62 0.52
query33 3.22 3.21 3.26
query34 15.47 4.01 3.40
query35 3.34 3.32 3.33
query36 0.53 0.44 0.43
query37 0.10 0.07 0.06
query38 0.06 0.03 0.03
query39 0.03 0.04 0.03
query40 0.17 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95.14 s
Total hot run time: 24.27 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6101091600
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68897:
URL: https://github.com/apache/doris/pull/68897#discussion_r4240634345
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1553,15 +1562,17 @@ static FileStoreTable dropCatalogLoader(FileStoreTable dataTable) {
if (dataTable.catalogEnvironment().catalogLoader() == null) {
return dataTable;
}
- FileStoreTable undecorated = PaimonTableDecorators.unwrapToFallbackOrBase(dataTable);
- if (undecorated instanceof FallbackReadFileStoreTable) {
- FallbackReadFileStoreTable fallbackReadTable = (FallbackReadFileStoreTable) undecorated;
- return new FallbackReadFileStoreTable(
- rebuildWithoutCatalogLoader(fallbackReadTable.wrapped()),
- rebuildWithoutCatalogLoader(fallbackReadTable.other()),
- PaimonReaderOptions.isWrappedFirst(fallbackReadTable));
+ return dropCatalogLoaderOfBranches(PaimonTableDecorators.unwrapToFallbackOrBase(dataTable));
+ }
+
+ private static FileStoreTable dropCatalogLoaderOfBranches(FileStoreTable table) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ // A chain table nests its snapshot and delta branches in a pair of their own; rebuild those too.
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable) table;
+ return PaimonTableDecorators.rebuildFallbackPair(pair,
+ dropCatalogLoaderOfBranches(pair.wrapped()), dropCatalogLoaderOfBranches(pair.other()));
}
Review Comment:
Confirmed, but it is not specific to this PR or to chain tables, so not changed here.
The side-channel source has never carried the catalog pin, for any branch. The FE sends `doris.serialized-system-source` for every system table that has a source (`backendManifestParallelismCap` always returns a value), and the BE then always rebuilds the system table from that source (`PaimonJniScanner.applyBackendManifestParallelism`), so the `pinCatalogSnapshot` applied to `serializedTable` never reaches the BE's plan. On master that already means a `$partitions` or `$files` read in a version-managed catalog plans every branch, main and snapshot alike, on the newest snapshot in storage rather than on the catalog pointer; this has been so since #66247 added the side channel. This PR only adds the delta leaf to that source. Before it, the flattened payload had no delta leaf at all, so `$partitions` never listed a delta-only partition, published or not, which is the larger error.
The fix for all of these is to pin the side-channel source the way `buildBackendTable` pins `serializedTable`: before the relation's `@options` are applied, so that a user's time travel still wins. That changes every system-table read in a version-managed catalog, so it should be its own PR with its own tests rather than part of this chain-table one.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -778,6 +779,11 @@ private static Table applyManifestParallelismBound(
}
// Each branch owns an independent planner setting; a smaller sibling is not an
// execution ceiling and must never throttle the other branch.
+ if (pair instanceof ChainGroupReadTable) {
+ // A chain table nests this group over its snapshot and delta branches. The group plans
Review Comment:
Confirmed, and not changed: in that window the BE now fails the way Paimon itself fails on the same table.
With this PR the BE reads `$files` through the table Paimon builds for a chain table. `FilesTable.FilesRead.createReader` replans each marker on that table and casts every split it gets back to `DataSplit`, and the fallback pair returns the chain split for a partition its main branch no longer has, so Spark or Flink reading `t$files` in the same window would raise the same `ClassCastException`.
The window is the two-phase `$files` gap the `pinCatalogSnapshot` javadoc already records: the FE takes its markers from the latest snapshot, and the BE replans later. With master's flattened payload the same race did not fail, but it returned the snapshot branch's files under a partition the FE had just attributed to main, that is, rows of another branch. An error is the better outcome there, and the race needs main to drop a partition between planning and the read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68897:
URL: https://github.com/apache/doris/pull/68897#issuecomment-6140185422
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org