You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "suxiaogang223 (via GitHub)" <gi...@apache.org> on 2026/10/09 07:52:39 UTC
[PR] [feature](paimon) Support Paimon Rust reader on master [doris]
suxiaogang223 opened a new pull request, #68825:
URL: https://github.com/apache/doris/pull/68825
### What problem does this PR solve?
Issue Number: None
Related PR: #66227
Problem Summary:
Forward-port the Paimon Rust reader from branch-4.1 to master. The reader consumes FE-planned Paimon splits in BE through the Arrow C Data Interface and is guarded by the `enable_paimon_rust_reader` session variable, which remains disabled by default.
The forward-port is adapted to the master connector framework instead of copying the branch-4.1 FE scan path. FE selects the Rust reader through the Paimon connector scan provider and checks BE capability advertised by heartbeat. Unsupported table layouts, scan modes, credentials, schemas, projections, predicates, or runtime filters conservatively fall back to the existing JNI reader.
Master-specific compatibility fixes include:
- preserve null-aware predicate and runtime-filter semantics;
- reject unsupported credential combinations without changing effective catalog credentials in tests;
- follow master branch and snapshot resolution behavior;
- carry name-based nested projection state even when a connector has no stable nested field IDs, preventing projected Arrow structs from being decoded as complete Doris structs;
- explicitly enable TIMESTAMP_LTZ mapping in the time-zone regression fixture.
### Release note
Add the experimental `enable_paimon_rust_reader` session variable. When enabled and supported by the selected BE and table scan, Doris can read Paimon data through the BE-side Rust reader. Unsupported scans automatically use the existing JNI reader.
### Check List (For Author)
- Test:
- FE build and checkstyle
- FE unit tests: `PaimonRustReaderSelectorTest`, `PluginDrivenScanNodeProjectionTest`
- Regression tests: `test_paimon_rust_reader_cast_predicates`, `test_paimon_rust_reader_schema_evolution`, `test_paimon_rust_reader_compatibility`, `test_paimon_rust_reader_eq_for_null`
- Behavior changed: Yes. The new reader can be enabled explicitly; the default remains unchanged.
- Does this need documentation: Yes. Session-variable documentation should be added with the feature rollout.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6162507196
I compared this PR at `13e491640aa` with branch-4.1 #66227 while working on bringing branch-4.x changes back to master, mostly by reading the code. The CI results below are from this head; when I wrote this, BE UT had passed, External had two failures and was still running, and FE UT and NonConcurrent were still running. The BE reader, the predicate converter and the BE unit tests match branch-4.1 closely, and every FE gate of `PaimonScanNode` has a counterpart in `PaimonRustReaderSelector` / `PaimonRustReaderCapabilities`. 13e491640aa also fixed the sliced Arrow batches: the row range is now read from the original batch. The points below are where it still differs.
**1. The Performance build does not compile.** GCC stops at `paimon_rust_table_reader.cpp:142` with `-Werror=subobject-linkage`: the members of `PaimonRustTableReader::PaimonHandles` use `table_ptr`, `read_builder_ptr` and the other aliases from the anonymous namespace. GCC does not report this for the main source file, but master compiles `Format` as a unity build, so the file is compiled inside `unity_7_cxx.cxx`. Performance is the only GCC build; it failed the same way at 19a6bab262e. Moving the deleters and aliases into a named namespace should fix it.
**2. With the Rust reader on, COUNT(col), MIN and MAX on append tables no longer use footer aggregates (13e491640aa).** Compatible splits now go to the Rust reader before the native check (`PaimonScanPlanProvider` L991-1002). branch-4.1 keeps raw-file splits on the native reader on purpose: "only the native file reader can use footer aggregates for COUNT(col) and MIN/MAX" (`PaimonScanNode` L1238-1241). Its `test_paimon_rust_reader_compatibility` runs the footer-aggregate queries with the Rust reader both on and off and asserts the native reader (L113-138). Here that block now runs only with the Rust reader off (L119-123), and the forced case that checked that the Rust reader returns real rows to the upper aggregate expects JNI (L147-148), so neither side is covered any more. Either keep raw-file splits ahead of the Rust reader as branch-4.1 does, or at least do so when the scan pushes down an aggregate; in both cases a Rust-enabled run of that block would catch a regression. Also,
Rust ranges are now planned before the `ignore_split_type` checks, so `IGNORE_JNI` no longer drops them as it does on branch-4.1.
**3. `test_paimon_rust_reader_eq_for_null` fails because the DDL mapping is not ported.** It fails at L222-223: expected `[[1, abc, 1.23], [2, xyz, 2.35]]`, but got `[[1, abcdef, 1.23], [2, xyzuvw, 2.35]]`. The Javadoc of `PaimonTypeMapping.toPaimonType` says CHAR/VARCHAR bounds and timestamp precision are preserved, but the code still maps CHAR(n) and VARCHAR(n) to VARCHAR(2147483647) and DATETIME to TIMESTAMP(6), and `PaimonTypeMappingToPaimonTest` still asserts that. So `MODIFY COLUMN v VARCHAR(3)` (L208) widens the column and Doris reads it back as STRING, which is also why `truncate_char_or_varchar_columns` (L220) does not help. I opened #69045 with the branch-4.1 mapping and its tests. Its Javadoc is the same text as yours, so the two PRs do not conflict. (The profile changes of #66227 are in #69044.)
**4. `test_paimon_rust_reader_compatibility` fails on the new routing.** The check before the ALTER at L156 expects no Rust reader, but `string_to_temporal_0` is an append table, which now goes to the Rust reader; branch-4.1 expects the Rust reader there too (L149). The suite stops at this failure, so the checks after it have not run. They may fail next: their baseline is now the native reader (`force_jni_scanner=false` at L47; branch-4.1 sets it to true, so its baseline is JNI), while the expected values are Java cast results (`1970-02-12`, `1970-01-01 00:00:00.042`, epoch seconds, and `127` for BIGINT to TINYINT).
**5. The session variable is never fuzzed.** `enable_paimon_rust_reader` has `fuzzy = true`, but `setFuzzyForCatalog` does not randomize it; branch-4.1 sets `this.enablePaimonRustReader = random.nextBoolean()` there. On master the annotation only affects printing, so CI never runs the Paimon suites with the Rust reader.
**6. The selector still rejects some scans that branch-4.1 reads with the Rust reader.** These only cost performance, but they would be worth stating in the description:
- `deletion-vectors.merge-on-read=true` sends every table to JNI. branch-4.1 only does that for partial-update and aggregation tables with deletion vectors, and its tests expect deduplicate + merge-on-read to use the Rust reader.
- The COUNT(*) range is always JNI.
- Any access path deeper than one level falls back.
- A TIMESTAMP(7..9) column anywhere in the current schema sends the whole table to JNI, even when it is not projected.
**7. FE tests.** branch-4.1 adds 73 FE test methods, and about 50 have no counterpart here. In particular, nothing covers the 45 schema-evolution, aggregation and primary-key cases of `PaimonRustReaderCapabilities`. The class is nearly identical to branch-4.1, so `PaimonRustReaderCompatibilityTest`, `PaimonRustReaderEvolutionTest`, `PaimonRustReaderLegacySchemaTest` and the capability cases of `PaimonScanNodeTest` should port with small changes. The Rust range payload (schema JSON without time-travel selectors, relation options, branch) has no FE test either.
One change goes the other way: the `EQ_FOR_NULL` change in `paimon_rust_predicate_converter.cpp` is a real fix. branch-4.1 pushes `IS NULL` for `col <=> non-null literal` and drops matching rows when `NULL_SAFE_EQUAL_TO_EQUAL` is disabled. `EqForNullLiteralIsPushed` only checks that some predicate is pushed, so it also passes with the branch-4.1 code; a test that reads rows would pin the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6096981820
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (3/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_701da299bb5dc73be36da3cf7e3346905c1250bd_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_701da299bb5dc73be36da3cf7e3346905c1250bd_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.71% (35268/45974) |
| Line Coverage | 61.89% (398778/644284) |
| Region Coverage | 58.36% (336873/577270) |
| Branch Coverage | 59.19% (154956/261811) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6076818232
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6080873304
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (3/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_5bfd0302d26a643cd19a6f418a4186372777b81d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_5bfd0302d26a643cd19a6f418a4186372777b81d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.68% (35246/45968) |
| Line Coverage | 61.87% (398605/644233) |
| Region Coverage | 58.30% (336492/577204) |
| Branch Coverage | 59.15% (154847/261787) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6189917597
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6187510399
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (3/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.66% (35940/46279) |
| Line Coverage | 63.12% (409161/648220) |
| Region Coverage | 60.15% (349467/580988) |
| Branch Coverage | 60.57% (159782/263812) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6166489204
<details>
<summary>ClickBench: <b>Total hot run time: 24.73 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8d52953c3037b078816cdc14d5695e371b5cfacf, data reload: false
query1 0.00 0.00 0.01
query2 0.10 0.05 0.05
query3 0.27 0.12 0.11
query4 1.61 0.12 0.12
query5 0.25 0.25 0.24
query6 1.16 0.69 0.69
query7 0.04 0.01 0.00
query8 0.06 0.04 0.04
query9 0.41 0.36 0.36
query10 0.59 0.61 0.58
query11 0.21 0.14 0.13
query12 0.19 0.15 0.15
query13 0.48 0.50 0.49
query14 1.02 1.02 1.02
query15 0.64 0.61 0.63
query16 0.34 0.34 0.38
query17 1.13 1.12 1.14
query18 0.24 0.25 0.22
query19 2.05 1.99 2.02
query20 0.02 0.02 0.01
query21 15.49 0.27 0.14
query22 4.40 0.06 0.05
query23 16.24 0.32 0.12
query24 3.21 0.73 0.41
query25 0.10 0.08 0.05
query26 1.55 0.25 0.15
query27 0.05 0.04 0.03
query28 1.96 0.60 0.42
query29 12.47 4.22 3.35
query30 0.30 0.14 0.13
query31 2.77 0.57 0.34
query32 3.23 0.61 0.52
query33 3.16 3.25 3.25
query34 15.64 4.01 3.40
query35 3.37 3.32 3.34
query36 0.57 0.43 0.43
query37 0.10 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.14 0.13
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 95.85 s
Total hot run time: 24.73 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6182963352
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6169982135
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6166236500
<details>
<summary>TPC-H: <b>Total hot run time: 29742 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8d52953c3037b078816cdc14d5695e371b5cfacf, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17585 5648 5558 5558
q2 2146 314 265 265
q3 10199 1435 859 859
q4 4680 475 347 347
q5 7443 800 547 547
q6 191 189 153 153
q7 775 804 625 625
q8 9257 1391 1115 1115
q9 5536 4287 4277 4277
q10 6832 1303 1006 1006
q11 431 260 242 242
q12 638 402 290 290
q13 18098 3654 3041 3041
q14 284 260 250 250
q15 q16 752 725 657 657
q17 1208 761 577 577
q18 7173 6249 6177 6177
q19 1212 964 645 645
q20 375 338 221 221
q21 6315 2582 2646 2582
q22 414 368 308 308
Total cold run time: 101544 ms
Total hot run time: 29742 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6724 6658 6753 6658
q2 688 602 575 575
q3 5542 5794 5320 5320
q4 2065 2206 1547 1547
q5 5831 5540 5879 5540
q6 241 184 144 144
q7 2278 1945 1816 1816
q8 3212 2799 2790 2790
q9 8135 8055 7816 7816
q10 3626 3614 3248 3248
q11 599 401 378 378
q12 664 691 503 503
q13 3283 3658 3010 3010
q14 286 285 257 257
q15 q16 651 685 617 617
q17 7899 6999 6864 6864
q18 13101 12245 13190 12245
q19 926 809 789 789
q20 2180 2159 1917 1917
q21 5816 4735 5010 4735
q22 508 448 402 402
Total cold run time: 74255 ms
Total hot run time: 67171 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6183560696
<details>
<summary>TPC-H: <b>Total hot run time: 29548 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b3e30fa9240264b19d185ee444c6531d4106a8ff, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17630 5688 5635 5635
q2 2098 311 252 252
q3 10262 1409 868 868
q4 4678 468 355 355
q5 7432 807 556 556
q6 198 195 158 158
q7 783 817 626 626
q8 9277 1355 1198 1198
q9 5659 4267 4307 4267
q10 6823 1297 1009 1009
q11 439 260 236 236
q12 633 404 283 283
q13 18104 3557 3035 3035
q14 281 278 254 254
q15 q16 729 729 679 679
q17 1205 757 582 582
q18 7096 6246 6197 6197
q19 1115 962 607 607
q20 400 352 224 224
q21 5418 2256 2437 2256
q22 370 315 271 271
Total cold run time: 100630 ms
Total hot run time: 29548 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6186 6121 6154 6121
q2 672 590 529 529
q3 5216 5381 4953 4953
q4 2003 2130 1556 1556
q5 5266 5467 5763 5467
q6 266 194 160 160
q7 2265 2094 1875 1875
q8 3255 2849 2786 2786
q9 8096 7947 8250 7947
q10 3850 3696 3360 3360
q11 621 452 424 424
q12 720 725 516 516
q13 3533 3921 3257 3257
q14 312 303 291 291
q15 q16 696 711 657 657
q17 8412 7505 6936 6936
q18 13196 12247 13078 12247
q19 930 812 803 803
q20 2201 2175 1943 1943
q21 5912 4802 4868 4802
q22 501 492 408 408
Total cold run time: 74109 ms
Total hot run time: 67038 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6081574159
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6184154264
# FE UT Coverage Report
Increment line coverage ` 75.86% (22/29)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6163261344
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (3/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.98% (35573/46208) |
| Line Coverage | 62.25% (402988/647392) |
| Region Coverage | 58.71% (340612/580130) |
| Branch Coverage | 59.56% (156854/263346) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6186486279
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6077634195
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6080541300
# FE UT Coverage Report
Increment line coverage ` 81.48% (22/27)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_5bfd0302d26a643cd19a6f418a4186372777b81d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_5bfd0302d26a643cd19a6f418a4186372777b81d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6095464494
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6163492087
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6166403825
<details>
<summary>TPC-DS: <b>Total hot run time: 153018 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8d52953c3037b078816cdc14d5695e371b5cfacf, data reload: false
query5 4309 610 468 468
query6 422 202 184 184
query7 4812 439 225 225
query8 319 177 173 173
query9 8763 4109 4085 4085
query10 460 312 265 265
query11 5954 3755 3530 3530
query12 151 92 86 86
query13 1276 459 343 343
query14 6157 4024 3775 3775
query14_1 3583 3586 3556 3556
query15 208 196 181 181
query16 947 432 420 420
query17 849 658 539 539
query18 2419 425 334 334
query19 194 171 137 137
query20 87 86 85 85
query21 210 138 112 112
query22 13023 12970 12887 12887
query23 13481 12918 12626 12626
query23_1 12590 12551 12549 12549
query24 7107 843 486 486
query24_1 484 492 474 474
query25 539 421 365 365
query26 1271 260 140 140
query27 2758 437 270 270
query28 4631 1988 1977 1977
query29 1569 594 465 465
query30 307 217 189 189
query31 998 927 806 806
query32 137 95 94 94
query33 537 308 249 249
query34 918 845 497 497
query35 746 752 644 644
query36 798 824 746 746
query37 137 100 92 92
query38 1782 1782 1701 1701
query39 786 801 740 740
query39_1 756 725 755 725
query40 221 113 97 97
query41 74 68 71 68
query42 89 86 87 86
query43 361 372 324 324
query44 1286 711 703 703
query45 181 184 172 172
query46 875 1008 569 569
query47 2807 3008 2931 2931
query48 301 305 216 216
query49 576 393 294 294
query50 669 275 212 212
query51 10505 10712 10540 10540
query52 75 82 65 65
query53 184 208 150 150
query54 240 189 179 179
query55 70 69 61 61
query56 223 257 223 223
query57 1614 1564 1552 1552
query58 284 266 256 256
query59 2058 2103 1930 1930
query60 274 241 225 225
query61 154 144 157 144
query62 389 338 286 286
query63 192 161 169 161
query64 2730 927 777 777
query65 3298 3227 3229 3227
query66 1786 408 311 311
query67 20164 20157 20082 20082
query68 3275 986 616 616
query69 406 282 244 244
query70 839 803 792 792
query71 306 228 204 204
query72 2793 2608 2273 2273
query73 530 524 295 295
query74 4614 4470 4325 4325
query75 2314 2283 1961 1961
query76 2382 1003 583 583
query77 330 368 270 270
query78 9740 9681 9175 9175
query79 956 843 524 524
query80 1214 414 358 358
query81 584 321 273 273
query82 611 131 105 105
query83 294 193 180 180
query84 310 116 101 101
query85 847 410 374 374
query86 390 253 235 235
query87 2026 1960 1833 1833
query88 3643 2728 2725 2725
query89 347 301 256 256
query90 1876 185 178 178
query91 164 147 126 126
query92 98 88 90 88
query93 1025 1016 590 590
query94 636 330 296 296
query95 574 391 317 317
query96 671 519 231 231
query97 2469 2430 2352 2352
query98 164 152 145 145
query99 786 755 643 643
Total cold run time: 235153 ms
Total hot run time: 153018 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6175021011
Thanks for the update. I re-read 446fd368cc1 against my earlier comment: points 1, 2, 4 and 5 are addressed, and most of the 4.x FE tests are now ported.
What remains:
- **Merge conflict with master.** #69045 has merged the same CHAR/VARCHAR length and DATETIME precision mapping (e771daa9b01), so `PaimonTypeMapping.java` and `PaimonTypeMappingToPaimonTest.java` conflict. Every TeamCity job on 446fd368cc1 stopped at `merge refs/pull/68825/head into master failed`, before compiling anything. Rebasing onto master and dropping this branch's copy of the mapping should unblock them.
- **Routing.** With an aggregate pushed down a split now stays native, as in 4.x. Without one, a split that can be read natively still goes to Rust first, while 4.x lets Rust replace only JNI.
- **Description.** The fallbacks that differ from 4.x (COUNT(*) ranges, access paths deeper than one level, TIMESTAMP(7..9)) are explained in the selector's Javadoc but not in the PR description; listing them there, with the routing difference above, would help reviewers.
- **Tests still without a counterpart** (by name, against 4.x): `testPrimaryKeyMergeEnginesWithOneInsertOnlyFile`, the query-auth, REST token and environment credential provider cases, the mixed-format ORC LTZ case, and the primary-key deletion-vector level-0 / delete matrix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6169990569
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6163410656
# FE Regression Coverage Report
Increment line coverage ` 12.08% (18/149)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6187658819
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (3/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.66% (35940/46279) |
| Line Coverage | 63.12% (409161/648220) |
| Region Coverage | 60.15% (349467/580988) |
| Branch Coverage | 60.57% (159782/263812) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6143849699
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6157792705
# FE Regression Coverage Report
Increment line coverage ` 66.67% (18/27)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_19a6bab262e26ece93dc9bc5510af5c4827aa6fd_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_19a6bab262e26ece93dc9bc5510af5c4827aa6fd_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6185576017
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (3/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.66% (35940/46279) |
| Line Coverage | 63.12% (409161/648220) |
| Region Coverage | 60.15% (349467/580988) |
| Branch Coverage | 60.57% (159782/263812) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 merged PR #68825:
URL: https://github.com/apache/doris/pull/68825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6187885515
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (3/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.97% (35623/46279) |
| Line Coverage | 62.25% (403501/648220) |
| Region Coverage | 58.64% (340693/580988) |
| Branch Coverage | 59.56% (157121/263812) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6183674250
<details>
<summary>TPC-DS: <b>Total hot run time: 152887 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b3e30fa9240264b19d185ee444c6531d4106a8ff, data reload: false
query5 4316 598 460 460
query6 441 205 195 195
query7 4797 454 231 231
query8 324 168 170 168
query9 8737 4046 4033 4033
query10 464 320 266 266
query11 6057 3835 3529 3529
query12 150 105 89 89
query13 1270 467 325 325
query14 6094 4029 3731 3731
query14_1 3549 3558 3587 3558
query15 218 207 182 182
query16 945 448 405 405
query17 856 658 539 539
query18 2408 430 311 311
query19 196 168 139 139
query20 87 82 82 82
query21 212 138 124 124
query22 12946 12988 12878 12878
query23 13608 12917 12614 12614
query23_1 12617 12761 12673 12673
query24 7082 840 471 471
query24_1 470 471 479 471
query25 537 422 362 362
query26 1258 258 144 144
query27 2760 437 273 273
query28 4607 1962 1946 1946
query29 1579 603 467 467
query30 310 228 186 186
query31 1021 930 829 829
query32 140 97 96 96
query33 509 310 251 251
query34 894 857 504 504
query35 756 741 658 658
query36 834 825 716 716
query37 144 103 96 96
query38 1806 1751 1721 1721
query39 789 802 753 753
query39_1 751 737 741 737
query40 221 120 99 99
query41 78 69 69 69
query42 88 85 85 85
query43 364 382 326 326
query44 1304 702 712 702
query45 185 181 168 168
query46 846 964 570 570
query47 2921 3061 2908 2908
query48 300 297 216 216
query49 565 403 297 297
query50 692 274 215 215
query51 10438 10553 10493 10493
query52 77 80 68 68
query53 184 208 156 156
query54 237 211 175 175
query55 71 64 60 60
query56 241 208 214 208
query57 1616 1595 1556 1556
query58 290 269 252 252
query59 2113 2079 1906 1906
query60 294 245 220 220
query61 148 140 148 140
query62 396 345 293 293
query63 215 161 161 161
query64 2778 953 811 811
query65 3292 3263 3249 3249
query66 1789 413 311 311
query67 20300 20137 20196 20137
query68 2902 999 604 604
query69 385 288 255 255
query70 881 768 771 768
query71 300 239 206 206
query72 2729 2602 2274 2274
query73 510 528 296 296
query74 4606 4442 4309 4309
query75 2357 2294 1941 1941
query76 2303 1009 598 598
query77 337 393 303 303
query78 9695 9671 9102 9102
query79 989 891 510 510
query80 547 432 346 346
query81 544 317 271 271
query82 572 136 108 108
query83 303 205 182 182
query84 322 118 103 103
query85 823 423 379 379
query86 325 238 228 228
query87 2012 1943 1822 1822
query88 3608 2712 2693 2693
query89 342 293 259 259
query90 1895 184 178 178
query91 167 149 128 128
query92 103 105 91 91
query93 1014 1003 579 579
query94 465 309 288 288
query95 593 395 318 318
query96 658 520 235 235
query97 2464 2425 2359 2359
query98 164 159 144 144
query99 781 774 652 652
Total cold run time: 234011 ms
Total hot run time: 152887 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6159981959
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6157575732
# FE UT Coverage Report
Increment line coverage ` 81.48% (22/27)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_19a6bab262e26ece93dc9bc5510af5c4827aa6fd/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_19a6bab262e26ece93dc9bc5510af5c4827aa6fd/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6168629502
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (3/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_8d52953c3037b078816cdc14d5695e371b5cfacf_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_8d52953c3037b078816cdc14d5695e371b5cfacf_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.88% (35571/46271) |
| Line Coverage | 62.16% (402851/648106) |
| Region Coverage | 58.41% (339323/580884) |
| Branch Coverage | 59.43% (156770/263769) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6183731801
<details>
<summary>ClickBench: <b>Total hot run time: 24.62 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b3e30fa9240264b19d185ee444c6531d4106a8ff, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.05
query3 0.27 0.12 0.12
query4 1.61 0.13 0.12
query5 0.28 0.25 0.22
query6 1.18 0.67 0.68
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.40 0.37 0.33
query10 0.57 0.60 0.58
query11 0.21 0.14 0.14
query12 0.18 0.14 0.14
query13 0.49 0.50 0.50
query14 1.02 1.02 1.01
query15 0.63 0.63 0.61
query16 0.33 0.34 0.35
query17 1.14 1.16 1.08
query18 0.22 0.21 0.21
query19 2.16 2.03 1.99
query20 0.02 0.02 0.01
query21 15.50 0.27 0.14
query22 4.55 0.06 0.05
query23 16.27 0.32 0.13
query24 2.93 0.62 0.41
query25 0.11 0.07 0.05
query26 1.12 0.25 0.14
query27 0.04 0.03 0.03
query28 1.61 0.62 0.41
query29 12.54 4.22 3.33
query30 0.29 0.14 0.13
query31 2.77 0.57 0.34
query32 3.22 0.60 0.52
query33 3.26 3.21 3.25
query34 15.46 4.03 3.42
query35 3.35 3.36 3.35
query36 0.56 0.44 0.42
query37 0.10 0.07 0.07
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.17 0.14 0.15
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95 s
Total hot run time: 24.62 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68825:
URL: https://github.com/apache/doris/pull/68825#discussion_r4280224754
##########
be/src/format_v2/table/paimon_rust_predicate_converter.cpp:
##########
@@ -0,0 +1,837 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+
+#include <algorithm>
+#include <cctype>
+#include <limits>
+#include <memory>
+#include <utility>
+
+#include "common/logging.h"
+#include "core/column/column_const.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "core/types.h"
+#include "core/value/decimalv2_value.h"
+#include "core/value/timestamptz_value.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vcompound_pred.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vectorized_fn_call.h"
+#include "exprs/vexpr.h"
+#include "exprs/vin_predicate.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+
+namespace doris {
+
+namespace {
+// paimon_datum tags (see paimon.h / bindings/c/src/table.rs::datum_from_c).
+constexpr int32_t kTagBool = 0;
+constexpr int32_t kTagTinyInt = 1;
+constexpr int32_t kTagSmallInt = 2;
+constexpr int32_t kTagInt = 3;
+constexpr int32_t kTagLong = 4;
+constexpr int32_t kTagDouble = 6;
+constexpr int32_t kTagString = 7;
+constexpr int32_t kTagDate = 8;
+constexpr int32_t kTagTimestamp = 10;
+constexpr int32_t kTagDecimal = 12;
+constexpr int32_t kTagBytes = 13;
+
+// paimon decimal precision ceiling (paimon::Decimal::MAX_PRECISION).
+constexpr int32_t kPaimonDecimalMaxPrecision = 38;
+
+// RAII for an owned paimon_predicate*. and/or/not consume their inputs, so we
+// release() before handing pointers to them.
+struct predicate_deleter {
+ void operator()(paimon_predicate* p) const {
+ if (p) {
+ paimon_predicate_free(p);
+ }
+ }
+};
+using predicate_ptr = std::unique_ptr<paimon_predicate, predicate_deleter>;
+
+// RAII for an owned paimon_error*.
+struct error_deleter {
+ void operator()(paimon_error* p) const {
+ if (p) {
+ paimon_error_free(p);
+ }
+ }
+};
+using error_ptr = std::unique_ptr<paimon_error, error_deleter>;
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_predicate_error(paimon_error* err) {
+ error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+} // namespace
+
+PaimonRustPredicateConverter::PaimonRustPredicateConverter(
+ const std::vector<std::string>& column_names, const std::vector<DataTypePtr>& column_types,
+ const paimon_table* table)
+ : _table(table) {
+ DORIS_CHECK(column_names.size() == column_types.size());
+ _columns_by_name.reserve(column_names.size());
+ for (size_t i = 0; i < column_names.size(); ++i) {
+ _columns_by_name.emplace(_normalize_name(column_names[i]),
+ std::make_pair(column_names[i], column_types[i]));
+ }
+ // Paimon TIMESTAMP (wall clock) is stored as epoch-millis-of-the-wall-time
+ // and the DateTimeV2 serde decodes timezone-naive arrow values in UTC, so
+ // timestamp literals convert wall->epoch in UTC. utc_time_zone() needs no
+ // tzdata lookup, so the conversion cannot silently fall back to a
+ // machine-local zone.
+ _utc_tz = cctz::utc_time_zone();
+}
+
+paimon_predicate* PaimonRustPredicateConverter::build(const VExprContextSPtrs& conjuncts) {
+ _converted_conjuncts = 0;
+ _converted_runtime_filters = 0;
+ if (_table == nullptr) {
+ return nullptr;
+ }
+ predicate_ptr result;
+ for (const auto& conjunct : conjuncts) {
+ if (!conjunct || !conjunct->root()) {
+ continue;
+ }
+ auto root = conjunct->root();
+ const bool is_runtime_filter = root->is_rf_wrapper();
+ if (root->is_rf_wrapper()) {
+ if (auto impl = root->get_impl()) {
+ // A null-aware runtime filter (an EQ_FOR_NULL join) must stay
+ // residual: its wrapper execution restores NULL probe rows to
+ // true (RuntimeFilterExpr::change_null_to_true), while the
+ // unwrapped impl — rebuilt as an ordinary IN predicate through
+ // VDirectInPredicate::get_slot_in_expr — treats NULL as
+ // not-in-set and would prune the NULL probes permanently
+ // before the join sees them. Keep the wrapper itself: it fails
+ // every dispatch below, so the conjunct stays in the residual,
+ // and it is safe to execute on selected rows, so later
+ // conjuncts keep pushing. The lance pushdown declines
+ // is_null_aware() filters for the same reason.
+ // is_null_aware() is concrete on RuntimeFilterExpr (the only
+ // class whose is_rf_wrapper() is true), so the dynamic_cast
+ // never fails in practice; the null guard keeps the unwrap for
+ // any future wrapper shape.
+ auto* rf_wrapper = dynamic_cast<RuntimeFilterExpr*>(root.get());
+ if (rf_wrapper == nullptr || !rf_wrapper->is_null_aware()) {
+ root = impl;
+ }
+ }
+ }
+ // Preserve a safe prefix of the conjunct order: a later pushed
+ // predicate (e.g. an arrived IN runtime filter) could otherwise prune
+ // rows on which an earlier error-preserving conjunct —
+ // assert_true(...), a failing cast, ... — must still raise. The v1
+ // partition-pruning path (FileScanner::_init_runtime_filter_partition_
+ // prune_ctxs) stops at is_safe_to_execute_on_selected_rows() for the
+ // same reason, so a convertible predicate after an unsafe conjunct
+ // must not be pushed. Safe conjuncts that cannot be converted keep
+ // the old skip: they cannot raise, so pruning rows before they are
+ // evaluated as the residual never loses an error.
+ if (!_is_safe_to_push(root)) {
+ break;
+ }
+ predicate_ptr pred(_convert_expr(root));
+ if (!pred) {
+ continue;
+ }
+ ++_converted_conjuncts;
+ _converted_runtime_filters += is_runtime_filter;
+ if (!result) {
+ result = std::move(pred);
+ } else {
+ // and consumes both inputs regardless of success.
+ result.reset(paimon_predicate_and(result.release(), pred.release()));
+ if (!result) {
+ _converted_conjuncts = 0;
+ _converted_runtime_filters = 0;
+ return nullptr;
+ }
+ }
+ }
+ return result.release();
+}
+
+bool PaimonRustPredicateConverter::_is_safe_to_push(const VExprSPtr& expr) {
+ if (expr->is_safe_to_execute_on_selected_rows()) {
+ return true;
+ }
+ // VectorizedFnCall::is_safe_to_execute_on_selected_rows() admits a fixed
+ // whitelist that does not include `like`, so a like conjunct would always
+ // stop the safe prefix here and never reach _convert_like. A like call
+ // whose children are themselves safe is just as total as the whitelisted
+ // comparisons: it only compares strings, and its regex is built by escaping
+ // the pattern operand (FunctionLike::convert_like_pattern), so no scanned
+ // value can make it raise — a malformed pattern or escape operand fails
+ // at open() on every execution, regardless of which rows survive. Admit
+ // it here rather than widening the generic whitelist, which gates shared
+ // BE pushdown paths outside this reader's scope.
+ auto* fn = dynamic_cast<VectorizedFnCall*>(expr.get());
+ if (fn == nullptr || _normalize_name(fn->function_name()) != "like") {
+ return false;
+ }
+ for (uint16_t i = 0; i < fn->get_num_children(); ++i) {
+ if (!fn->get_child(i)->is_safe_to_execute_on_selected_rows()) {
+ return false;
+ }
+ }
+ return true;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_expr(const VExprSPtr& expr) {
+ if (!expr) {
+ return nullptr;
+ }
+
+ // Casts are not unwrapped anywhere (predicate root included): a cast node
+ // fails every dispatch below and the conjunct stays in the Doris residual,
+ // mirroring the FE converter, which keeps casted expressions unconverted.
+ if (auto* direct_in = dynamic_cast<VDirectInPredicate*>(expr.get())) {
+ VExprSPtr in_expr;
+ if (direct_in->get_slot_in_expr(in_expr)) {
Review Comment:
[P2] Reject residual-only IN filters before expanding the set. An arrived IN runtime filter reaches this branch as VDirectInPredicate, and get_slot_in_expr walks its entire HybridSet and allocates one VLiteral per key. Only afterward does _convert_in reject oversized lists or unsupported slot types. With runtime_filter_max_in_num=70000, the 65535-key case added in this PR therefore creates and discards 65535 literals for every logical split while pushing nothing; 10000 splits cause over 655 million unnecessary literal allocations. Check the original set size and slot support before expanding it (the native ORC path already does the size check), and cover the actual VDirectInPredicate path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6188122753
# FE Regression Coverage Report
Increment line coverage ` 20.00% (19/95)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6150399285
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6163166502
# FE UT Coverage Report
Increment line coverage ` 81.48% (22/27)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_13e491640aa8d4bb03224c0190f1c40ec26b9db1/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6168591763
# FE Regression Coverage Report
Increment line coverage ` 41.30% (19/46)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_8d52953c3037b078816cdc14d5695e371b5cfacf_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_8d52953c3037b078816cdc14d5695e371b5cfacf_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68825:
URL: https://github.com/apache/doris/pull/68825#discussion_r4272204393
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonRustReaderSelector.java:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.rest.RESTTokenFileIO;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.MultisetType;
+import org.apache.paimon.types.RowType;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Chooses the Rust reader only for table, split, storage and backend shapes it can read safely.
+ * Nested pruning and precision 7-9 timestamps still require JNI because the pinned Arrow ABI
+ * cannot preserve their full semantics. COUNT(*) metadata ranges bypass this selector entirely.
+ */
+final class PaimonRustReaderSelector {
+ static final String BACKEND_CAPABILITY = "paimon-rust-reader";
+ private static final String ENABLE_PAIMON_RUST_READER = "enable_paimon_rust_reader";
+ private static final String ENABLE_FILE_SCANNER_V2 = "enable_file_scanner_v2";
+ private static final String FORCE_JNI_SCANNER = "force_jni_scanner";
+ private static final String DELETION_VECTORS_MERGE_ON_READ = "deletion-vectors.merge-on-read";
+ private static final Set<String> VERIFIED_LOCATION_SCHEMES =
+ new HashSet<>(Arrays.asList("s3", "s3a", "oss", "hdfs", "file"));
+ private static final Set<String> VERIFIED_HDFS_OPTION_KEYS = new HashSet<>(Arrays.asList(
+ "fs.defaultFS",
+ "hadoop.security.authentication",
+ "hdfs.security.authentication",
+ "ipc.client.fallback-to-simple-auth-allowed"));
+
+ private final FileStoreTable table;
+ private final PaimonRustReaderCapabilities capabilities;
+ private final Map<String, String> backendStorageProperties;
+ private final boolean scanCompatible;
+ private final boolean partialUpdateOrAggregateWithDeletionVectors;
+ private final boolean deletionVectorsMergeOnRead;
+
+ PaimonRustReaderSelector(ConnectorSession session, boolean backendsSupportRust,
+ FileStoreTable table, List<ConnectorColumnHandle> columns,
+ Map<String, String> backendStorageProperties, boolean usesFallbackRead,
+ boolean incrementalRead, boolean projectedVariant) {
+ this.table = table;
+ this.backendStorageProperties = backendStorageProperties;
+ CoreOptions options = table.coreOptions();
+ CoreOptions.MergeEngine mergeEngine = options.mergeEngine();
+ partialUpdateOrAggregateWithDeletionVectors = options.deletionVectorsEnabled()
+ && (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+ || mergeEngine == CoreOptions.MergeEngine.AGGREGATE);
+ TableSchema schema = table.schema();
+ String mergeOnRead = schema.options().get(DELETION_VECTORS_MERGE_ON_READ);
+ deletionVectorsMergeOnRead = "true".equalsIgnoreCase(mergeOnRead);
+
+ String location = table.location().toString();
+ boolean enabled = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(ENABLE_PAIMON_RUST_READER));
+ boolean fileScannerV2 = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(ENABLE_FILE_SCANNER_V2));
+ boolean forceJniScanner = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(FORCE_JNI_SCANNER));
+ boolean restTokenTable = table.fileIO() instanceof RESTTokenFileIO;
+ boolean unsupportedMergeOption = (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+ || mergeEngine == CoreOptions.MergeEngine.AGGREGATE)
+ && hasRustUnsupportedMergeOption(schema.options(), mergeEngine);
+ boolean deduplicateIgnoreDelete = mergeEngine == CoreOptions.MergeEngine.DEDUPLICATE
+ && options.ignoreDelete();
+ String scanMode = PaimonScanParams.withoutTimeTravelSelectors(schema)
+ .options().get(CoreOptions.SCAN_MODE.key());
+ boolean scanModeSupported = scanMode == null || "default".equalsIgnoreCase(scanMode);
+
+ scanCompatible = enabled && fileScannerV2 && !forceJniScanner
+ && backendsSupportRust
+ && !usesFallbackRead && !incrementalRead && !projectedVariant
+ && !options.queryAuthEnabled() && !restTokenTable
+ && !(partialUpdateOrAggregateWithDeletionVectors && deletionVectorsMergeOnRead)
+ && !deduplicateIgnoreDelete
+ && !unsupportedMergeOption && scanModeSupported
+ && isRustVerifiedLocationScheme(location)
+ && (!isHdfsLocationScheme(location)
+ || isRustVerifiedHdfsBackend(backendStorageProperties))
+ && isCredentialProviderTranslatable(location, backendStorageProperties);
+ capabilities = new PaimonRustReaderCapabilities(table, columns);
+ }
+
+ boolean canRead(DataSplit split) {
+ if (!scanCompatible || splitHasExternalFiles(split)) {
+ return false;
+ }
+ if (splitHasOrcFile(split) && table.schema().fields().stream()
+ .anyMatch(field -> containsTimestampLtz(field.type()))) {
+ return false;
+ }
+ if (partialUpdateOrAggregateWithDeletionVectors && !deletionVectorsMergeOnRead
+ && !isFullyMaterializedPkDvSplit(split)) {
+ return false;
+ }
+ return capabilities.canRead(split);
+ }
+
+ static boolean isRustVerifiedLocationScheme(String location) {
+ if (location == null) {
+ return false;
+ }
+ int separator = location.indexOf("://");
Review Comment:
[P2] Reject or qualify authority-less HDFS table locations. A filesystem catalog with warehouse=hdfs:///user/paimon and fs.defaultFS=hdfs://namenode:8020 is valid for Java, but Paimon Path.toString emits hdfs:/user/paimon/db.db/t. Because this helper only looks for ://, it admits that path as scheme-less and isHdfsLocationScheme also skips its checks. RustScanMetadata sends the path unchanged; the pinned Rust HDFS parser requires a host (and hdfs:// for file paths) and does not translate fs.defaultFS, so otherwise eligible scans fail when Rust is enabled. Parse schemes independently of authority, then fully qualify HDFS paths or keep these scans on JNI, and cover this catalog form in selector/differential tests.
##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn) \
+ struct type##_deleter { \
+ void operator()(paimon_##type* p) const { \
+ if (p) { \
+ freefn(p); \
+ } \
+ } \
+ }; \
+ using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+ explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+ ~ArrowBatch() {
+ auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+ auto* array = static_cast<ArrowArray*>(batch_.array);
+ if (array && array->release) {
+ array->release(array);
+ }
+ if (schema && schema->release) {
+ schema->release(schema);
+ }
+ paimon_arrow_batch_free(batch_);
+ }
+
+ ArrowBatch(const ArrowBatch&) = delete;
+ ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+ ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+ ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+ paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+ rust_handles::error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+ std::string out;
+ for (const auto& kv : options) {
+ if (!out.empty()) {
+ out += ", ";
+ }
+ out += kv.first;
+ }
+ return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+ rust_handles::table_ptr table;
+ rust_handles::read_builder_ptr read_builder;
+ rust_handles::plan_ptr plan;
+ rust_handles::table_read_ptr table_read;
+ rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+ RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+ {
+ // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+ // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.init_timer);
+ // Materialize TIMESTAMP_LTZ in the session timezone — the same
+ // convention as the JNI reader (PaimonJniScanner reads time_zone from
+ // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+ // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+ // of _ctz, so NTZ wall-clock semantics are preserved.
+ DORIS_CHECK(_runtime_state != nullptr);
+ _ctz = _runtime_state->timezone_obj();
+ if (_scanner_profile != nullptr) {
+ file_scan_profile::ensure_hierarchy(_scanner_profile);
+ _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+ file_scan_profile::TABLE_READER);
+ _rust_open_split_time =
+ ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+ _rust_read_batch_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+ _rust_arrow_to_block_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+ _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_converted = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+ }
+ // Projected column name -> fixed output position, registered with both the exact and
+ // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+ // semantics: exact match first, lower-case fallback on lookup).
+ _output_name_to_idx.reserve(_projected_columns.size() * 2);
+ for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+ _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+ _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+ }
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+ // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+ // are idempotent, and clear it only when a new split is explicitly prepared.
+ _close_split_reader();
+ _split_eof = false;
+ _current_range = options.current_range;
+ RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+ if (current_split_pruned()) {
+ return Status::OK();
+ }
+ if (_is_table_level_count_active()) {
+ // No rust pipeline is opened; get_block emits the synthetic count rows.
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.prepare_split_timer);
+ // Open may dominate the scan or fail before get_block; include it in the parent timer.
+ SCOPED_TIMER(_rust_total_time);
+ SCOPED_TIMER(_rust_open_split_time);
+ RETURN_IF_ERROR(_open_split_reader(options.current_range));
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.exec_timer);
+ SCOPED_TIMER(_rust_total_time);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK(eos != nullptr);
+ DORIS_CHECK(block->columns() == _projected_columns.size());
+ block->clear_column_data(_projected_columns.size());
+ *eos = false;
+
+ if (_is_table_level_count_active()) {
+ return _read_table_level_count(block, eos);
+ }
+
+ // num_splits == 0 yields an empty (but valid) stream: report EOF.
+ if (_split_eof) {
+ *eos = true;
+ return Status::OK();
+ }
+ if (!_handles || !_handles->reader) {
+ return Status::InternalError("paimon-rust reader is not initialized");
+ }
+
+ while (true) {
+ // Mirror the base TableReader cancellation contract so a cancelled query does not
+ // drain the whole split.
+ if (_io_ctx != nullptr && _io_ctx->should_stop) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ if (!_pending_record_batch) {
+ paimon_result_next_batch next;
+ {
+ SCOPED_TIMER(_rust_read_batch_time);
+ next = paimon_record_batch_reader_next(_handles->reader.get());
Review Comment:
[P1] Charge Rust read buffers to the query memory tracker. The buffers produced by this call remain Rust-owned, and the zero-copy import retains them in _pending_record_batch. In this build malloc/posix_memalign only forward to jemalloc; query tracking is explicit in Doris Allocator, which the Rust system allocator bypasses. Neither this reader nor the C binding reserves or charges those bytes, including Parquet row-group/decode buffers and Tokio worker allocations. Even the added wide-string fixture holds roughly 4 MiB in the Rust batch alongside its tracked Doris copy at a 32-row cap. Large row groups or concurrent scanners can therefore exceed query/workload-group limits while only output columns are counted. Add a query-owned accounting/reservation bridge with balanced release on EOF, error, abort and close; output slicing alone does not address this separate issue.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonRustReaderSelector.java:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.rest.RESTTokenFileIO;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.MultisetType;
+import org.apache.paimon.types.RowType;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Chooses the Rust reader only for table, split, storage and backend shapes it can read safely.
+ * Nested pruning and precision 7-9 timestamps still require JNI because the pinned Arrow ABI
+ * cannot preserve their full semantics. COUNT(*) metadata ranges bypass this selector entirely.
+ */
+final class PaimonRustReaderSelector {
+ static final String BACKEND_CAPABILITY = "paimon-rust-reader";
+ private static final String ENABLE_PAIMON_RUST_READER = "enable_paimon_rust_reader";
+ private static final String ENABLE_FILE_SCANNER_V2 = "enable_file_scanner_v2";
+ private static final String FORCE_JNI_SCANNER = "force_jni_scanner";
+ private static final String DELETION_VECTORS_MERGE_ON_READ = "deletion-vectors.merge-on-read";
+ private static final Set<String> VERIFIED_LOCATION_SCHEMES =
+ new HashSet<>(Arrays.asList("s3", "s3a", "oss", "hdfs", "file"));
+ private static final Set<String> VERIFIED_HDFS_OPTION_KEYS = new HashSet<>(Arrays.asList(
+ "fs.defaultFS",
+ "hadoop.security.authentication",
+ "hdfs.security.authentication",
+ "ipc.client.fallback-to-simple-auth-allowed"));
+
+ private final FileStoreTable table;
+ private final PaimonRustReaderCapabilities capabilities;
+ private final Map<String, String> backendStorageProperties;
+ private final boolean scanCompatible;
+ private final boolean partialUpdateOrAggregateWithDeletionVectors;
+ private final boolean deletionVectorsMergeOnRead;
+
+ PaimonRustReaderSelector(ConnectorSession session, boolean backendsSupportRust,
+ FileStoreTable table, List<ConnectorColumnHandle> columns,
+ Map<String, String> backendStorageProperties, boolean usesFallbackRead,
+ boolean incrementalRead, boolean projectedVariant) {
+ this.table = table;
+ this.backendStorageProperties = backendStorageProperties;
+ CoreOptions options = table.coreOptions();
+ CoreOptions.MergeEngine mergeEngine = options.mergeEngine();
+ partialUpdateOrAggregateWithDeletionVectors = options.deletionVectorsEnabled()
+ && (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+ || mergeEngine == CoreOptions.MergeEngine.AGGREGATE);
+ TableSchema schema = table.schema();
+ String mergeOnRead = schema.options().get(DELETION_VECTORS_MERGE_ON_READ);
+ deletionVectorsMergeOnRead = "true".equalsIgnoreCase(mergeOnRead);
+
+ String location = table.location().toString();
+ boolean enabled = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(ENABLE_PAIMON_RUST_READER));
+ boolean fileScannerV2 = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(ENABLE_FILE_SCANNER_V2));
+ boolean forceJniScanner = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(FORCE_JNI_SCANNER));
+ boolean restTokenTable = table.fileIO() instanceof RESTTokenFileIO;
+ boolean unsupportedMergeOption = (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+ || mergeEngine == CoreOptions.MergeEngine.AGGREGATE)
+ && hasRustUnsupportedMergeOption(schema.options(), mergeEngine);
+ boolean deduplicateIgnoreDelete = mergeEngine == CoreOptions.MergeEngine.DEDUPLICATE
+ && options.ignoreDelete();
+ String scanMode = PaimonScanParams.withoutTimeTravelSelectors(schema)
+ .options().get(CoreOptions.SCAN_MODE.key());
+ boolean scanModeSupported = scanMode == null || "default".equalsIgnoreCase(scanMode);
+
+ scanCompatible = enabled && fileScannerV2 && !forceJniScanner
+ && backendsSupportRust
+ && !usesFallbackRead && !incrementalRead && !projectedVariant
+ && !options.queryAuthEnabled() && !restTokenTable
+ && !(partialUpdateOrAggregateWithDeletionVectors && deletionVectorsMergeOnRead)
+ && !deduplicateIgnoreDelete
+ && !unsupportedMergeOption && scanModeSupported
+ && isRustVerifiedLocationScheme(location)
+ && (!isHdfsLocationScheme(location)
+ || isRustVerifiedHdfsBackend(backendStorageProperties))
+ && isCredentialProviderTranslatable(location, backendStorageProperties);
+ capabilities = new PaimonRustReaderCapabilities(table, columns);
+ }
+
+ boolean canRead(DataSplit split) {
+ if (!scanCompatible || splitHasExternalFiles(split)) {
+ return false;
+ }
+ if (splitHasOrcFile(split) && table.schema().fields().stream()
Review Comment:
[P2] Preserve the full range of low-precision ORC timestamps. This guard rejects only LTZ, so an append-only ORC table with current-schema TIMESTAMP(3) or TIMESTAMP(6) and a value such as 3000-01-01 still selects Rust. The pinned Paimon OrcFormatReader leaves orc-rust 0.8.0 at its default nanosecond precision; that decoder converts epoch nanoseconds to i64 and errors outside roughly 1677-2262, before any cast to the table's millisecond/microsecond unit. Enabling Rust therefore makes a valid scan fail although JNI can read it. This is a distinct remaining instance from the existing TIMESTAMP(7-9)/Parquet INT96 thread: the <=6 schema guard cannot cover it. Decode ORC at a suitable unit or conservatively route these timestamp schemas to JNI, including nested leaves, and add an out-of-range ORC parity case. The pinned [ORC builder contract](https://docs.rs/orc-rust/0.8.0/orc_rust/arrow_reader/struct.ArrowReaderBuilder.html#method.with_timestamp_precision) documents this default.
##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn) \
+ struct type##_deleter { \
+ void operator()(paimon_##type* p) const { \
+ if (p) { \
+ freefn(p); \
+ } \
+ } \
+ }; \
+ using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+ explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+ ~ArrowBatch() {
+ auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+ auto* array = static_cast<ArrowArray*>(batch_.array);
+ if (array && array->release) {
+ array->release(array);
+ }
+ if (schema && schema->release) {
+ schema->release(schema);
+ }
+ paimon_arrow_batch_free(batch_);
+ }
+
+ ArrowBatch(const ArrowBatch&) = delete;
+ ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+ ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+ ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+ paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+ rust_handles::error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+ std::string out;
+ for (const auto& kv : options) {
+ if (!out.empty()) {
+ out += ", ";
+ }
+ out += kv.first;
+ }
+ return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+ rust_handles::table_ptr table;
+ rust_handles::read_builder_ptr read_builder;
+ rust_handles::plan_ptr plan;
+ rust_handles::table_read_ptr table_read;
+ rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+ RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+ {
+ // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+ // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.init_timer);
+ // Materialize TIMESTAMP_LTZ in the session timezone — the same
+ // convention as the JNI reader (PaimonJniScanner reads time_zone from
+ // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+ // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+ // of _ctz, so NTZ wall-clock semantics are preserved.
+ DORIS_CHECK(_runtime_state != nullptr);
+ _ctz = _runtime_state->timezone_obj();
+ if (_scanner_profile != nullptr) {
+ file_scan_profile::ensure_hierarchy(_scanner_profile);
+ _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+ file_scan_profile::TABLE_READER);
+ _rust_open_split_time =
+ ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+ _rust_read_batch_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+ _rust_arrow_to_block_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+ _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_converted = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+ }
+ // Projected column name -> fixed output position, registered with both the exact and
+ // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+ // semantics: exact match first, lower-case fallback on lookup).
+ _output_name_to_idx.reserve(_projected_columns.size() * 2);
+ for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+ _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+ _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+ }
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+ // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+ // are idempotent, and clear it only when a new split is explicitly prepared.
+ _close_split_reader();
+ _split_eof = false;
+ _current_range = options.current_range;
+ RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+ if (current_split_pruned()) {
+ return Status::OK();
+ }
+ if (_is_table_level_count_active()) {
+ // No rust pipeline is opened; get_block emits the synthetic count rows.
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.prepare_split_timer);
+ // Open may dominate the scan or fail before get_block; include it in the parent timer.
+ SCOPED_TIMER(_rust_total_time);
+ SCOPED_TIMER(_rust_open_split_time);
+ RETURN_IF_ERROR(_open_split_reader(options.current_range));
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.exec_timer);
+ SCOPED_TIMER(_rust_total_time);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK(eos != nullptr);
+ DORIS_CHECK(block->columns() == _projected_columns.size());
+ block->clear_column_data(_projected_columns.size());
+ *eos = false;
+
+ if (_is_table_level_count_active()) {
+ return _read_table_level_count(block, eos);
+ }
+
+ // num_splits == 0 yields an empty (but valid) stream: report EOF.
+ if (_split_eof) {
+ *eos = true;
+ return Status::OK();
+ }
+ if (!_handles || !_handles->reader) {
+ return Status::InternalError("paimon-rust reader is not initialized");
+ }
+
+ while (true) {
+ // Mirror the base TableReader cancellation contract so a cancelled query does not
+ // drain the whole split.
+ if (_io_ctx != nullptr && _io_ctx->should_stop) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ if (!_pending_record_batch) {
+ paimon_result_next_batch next;
+ {
+ SCOPED_TIMER(_rust_read_batch_time);
+ next = paimon_record_batch_reader_next(_handles->reader.get());
+ }
+ if (next.error != nullptr) {
Review Comment:
[P2] Preserve missing-data-file status on the Rust path. Rust now precedes the native reader for eligible ordinary append-only splits. If a single-file Parquet split's data file disappears after planning, Rust's storage error is returned here as INTERNAL_ERROR. FileScannerV2 skips only NOT_FOUND, so ignore_not_found_file_in_external_table (true by default, and applicable to Paimon) stops working; the same split on the native route preserves ENOENT as NOT_FOUND and follows that policy. Preserve a distinguishable missing-data-file error through the Rust binding and map it to the scanner's expected status, without treating other IO/schema errors as missing files. Add native/Rust differential coverage with the file removed after planning and both ignore settings.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -897,6 +911,16 @@ && hasTrustworthyLimitAccounting(table)
}
}
+ boolean incrementalRead = pinnedOptions.containsKey("incremental-between")
+ || pinnedOptions.containsKey("incremental-between-timestamp");
+ PaimonRustReaderSelector rustReaderSelector = table instanceof FileStoreTable
+ && !requiresMetadataColumns
+ ? new PaimonRustReaderSelector(session, backendsSupportRust,
+ (FileStoreTable) table, columns, backendStorageProperties(),
Review Comment:
[P2] Check and transport the effective Paimon storage configuration. This selector sees only the typed backend map, but PaimonCatalogFactory.applyStorageConfig applies raw fs.s3a.* and paimon.s3.* overrides after that map when constructing Java FileIO. For example, canonical s3.access_key/secret_key can name pair A while fs.s3a.access.key/secret.key override it with readable pair B. A non-raw append-only data-evolution split with INT columns previously uses JNI and retains B in its serialized FileIO; it now passes Rust's static-key gate using A. The Rust payload never carries the raw overlays and its schema-json FileIO constructor does not recover them, so the scan uses A and fails access where JNI succeeds. Preserve the effective option precedence in the Rust transport/eligibility check, or fall back for untranslatable overrides; add a differential catalog case.
##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn) \
+ struct type##_deleter { \
+ void operator()(paimon_##type* p) const { \
+ if (p) { \
+ freefn(p); \
+ } \
+ } \
+ }; \
+ using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+ explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+ ~ArrowBatch() {
+ auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+ auto* array = static_cast<ArrowArray*>(batch_.array);
+ if (array && array->release) {
+ array->release(array);
+ }
+ if (schema && schema->release) {
+ schema->release(schema);
+ }
+ paimon_arrow_batch_free(batch_);
+ }
+
+ ArrowBatch(const ArrowBatch&) = delete;
+ ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+ ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+ ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+ paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+ rust_handles::error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+ std::string out;
+ for (const auto& kv : options) {
+ if (!out.empty()) {
+ out += ", ";
+ }
+ out += kv.first;
+ }
+ return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+ rust_handles::table_ptr table;
+ rust_handles::read_builder_ptr read_builder;
+ rust_handles::plan_ptr plan;
+ rust_handles::table_read_ptr table_read;
+ rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+ RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+ {
+ // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+ // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.init_timer);
+ // Materialize TIMESTAMP_LTZ in the session timezone — the same
+ // convention as the JNI reader (PaimonJniScanner reads time_zone from
+ // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+ // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+ // of _ctz, so NTZ wall-clock semantics are preserved.
+ DORIS_CHECK(_runtime_state != nullptr);
+ _ctz = _runtime_state->timezone_obj();
+ if (_scanner_profile != nullptr) {
+ file_scan_profile::ensure_hierarchy(_scanner_profile);
+ _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+ file_scan_profile::TABLE_READER);
+ _rust_open_split_time =
+ ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+ _rust_read_batch_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+ _rust_arrow_to_block_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+ _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_converted = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+ }
+ // Projected column name -> fixed output position, registered with both the exact and
+ // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+ // semantics: exact match first, lower-case fallback on lookup).
+ _output_name_to_idx.reserve(_projected_columns.size() * 2);
+ for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+ _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+ _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+ }
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+ // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+ // are idempotent, and clear it only when a new split is explicitly prepared.
+ _close_split_reader();
+ _split_eof = false;
+ _current_range = options.current_range;
+ RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+ if (current_split_pruned()) {
+ return Status::OK();
+ }
+ if (_is_table_level_count_active()) {
+ // No rust pipeline is opened; get_block emits the synthetic count rows.
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.prepare_split_timer);
+ // Open may dominate the scan or fail before get_block; include it in the parent timer.
+ SCOPED_TIMER(_rust_total_time);
+ SCOPED_TIMER(_rust_open_split_time);
+ RETURN_IF_ERROR(_open_split_reader(options.current_range));
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.exec_timer);
+ SCOPED_TIMER(_rust_total_time);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK(eos != nullptr);
+ DORIS_CHECK(block->columns() == _projected_columns.size());
+ block->clear_column_data(_projected_columns.size());
+ *eos = false;
+
+ if (_is_table_level_count_active()) {
+ return _read_table_level_count(block, eos);
+ }
+
+ // num_splits == 0 yields an empty (but valid) stream: report EOF.
+ if (_split_eof) {
+ *eos = true;
+ return Status::OK();
+ }
+ if (!_handles || !_handles->reader) {
+ return Status::InternalError("paimon-rust reader is not initialized");
+ }
+
+ while (true) {
+ // Mirror the base TableReader cancellation contract so a cancelled query does not
+ // drain the whole split.
+ if (_io_ctx != nullptr && _io_ctx->should_stop) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ if (!_pending_record_batch) {
+ paimon_result_next_batch next;
+ {
+ SCOPED_TIMER(_rust_read_batch_time);
+ next = paimon_record_batch_reader_next(_handles->reader.get());
+ }
+ if (next.error != nullptr) {
+ return Status::InternalError("paimon-rust read batch failed: {}",
+ consume_error(next.error));
+ }
+ // End of stream: both pointers are null.
+ if (next.batch.array == nullptr && next.batch.schema == nullptr) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ // RAII: the batch's Arrow release callbacks + container free run when
+ // `batch` leaves this scope, including on any early return.
+ ArrowBatch batch(next.batch);
+ arrow::Result<std::shared_ptr<arrow::RecordBatch>> import_result =
+ arrow::ImportRecordBatch(batch.array(), batch.schema());
+ if (!import_result.ok()) {
+ return Status::InternalError("failed to import paimon-rust arrow batch: {}",
+ import_result.status().message());
+ }
+ _pending_record_batch = std::move(import_result).ValueUnsafe();
+ _pending_record_batch_offset = 0;
+ if (_pending_record_batch->num_rows() == 0) {
+ _pending_record_batch.reset();
+ continue;
+ }
+ }
+
+ const size_t batch_size = static_cast<size_t>(
+ _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+ const size_t remaining = static_cast<size_t>(_pending_record_batch->num_rows()) -
+ _pending_record_batch_offset;
+ const size_t rows = std::min(batch_size, remaining);
+ RETURN_IF_ERROR(_fill_block_from_record_batch(_pending_record_batch, block,
+ _pending_record_batch_offset, rows));
+ _pending_record_batch_offset += rows;
+ if (_pending_record_batch_offset ==
+ static_cast<size_t>(_pending_record_batch->num_rows())) {
+ _pending_record_batch.reset();
+ _pending_record_batch_offset = 0;
+ }
+ _record_scan_rows(rows);
+ *eos = false;
+ return Status::OK();
+ }
+}
+
+Status PaimonRustTableReader::abort_split() {
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.close_timer);
+ _close_split_reader();
+ _split_eof = false;
+ }
+ return format::TableReader::abort_split();
+}
+
+#ifdef BE_TEST
+std::string PaimonRustTableReader::TEST_format_options(
+ const std::map<std::string, std::string>& options) {
+ return format_options(options);
+}
+
+std::map<std::string, std::string> PaimonRustTableReader::TEST_build_options(
+ TFileScanRangeParams* scan_params, const TFileRangeDesc& range) {
+ TFileScanRangeParams* previous_params = _scan_params;
+ TFileRangeDesc previous_range = _current_range;
+ _scan_params = scan_params;
+ _current_range = range;
+ std::map<std::string, std::string> options = _build_options();
+ _scan_params = previous_params;
+ _current_range = std::move(previous_range);
+ return options;
+}
+#endif
+
+Status PaimonRustTableReader::close() {
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.close_timer);
+ _close_split_reader();
+ _close_table();
+ }
+ return format::TableReader::close();
+}
+
+Status PaimonRustTableReader::_validate_rust_split(const TFileRangeDesc& range) const {
+ if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) {
+ return Status::InternalError(
+ "missing paimon_params for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ const auto& params = range.table_format_params.paimon_params;
+ if (!params.__isset.paimon_split || params.paimon_split.empty()) {
+ return Status::InternalError(
+ "missing paimon_split for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ if (params.__isset.reader_type && params.reader_type != TPaimonReaderType::PAIMON_RUST) {
+ return Status::InternalError(
+ "invalid reader_type for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ if (!_resolve_table_path(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing paimon_table; cannot resolve paimon table location");
+ }
+ if (!_resolve_db_name(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing db_name; cannot open paimon table via schema json");
+ }
+ if (!_resolve_table_name(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing table_name; cannot open paimon table via schema json");
+ }
+ if (!_resolve_table_schema_json(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing paimon_table_schema_json; cannot open paimon table via "
+ "schema json");
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_open_split_reader(const TFileRangeDesc& range) {
+ // 1. Decode the FE-planned split first so we fail fast (and without any
+ // filesystem IO) when it is missing or malformed.
+ std::string split_bytes;
+ RETURN_IF_ERROR(_decode_split_bytes(&split_bytes));
+
+ // 2. Resolve identifier + table_path + FE-supplied TableSchema JSON.
+ auto table_path = _resolve_table_path(range).value();
+ auto db_name = _resolve_db_name(range).value();
+ auto table_name = _resolve_table_name(range).value();
+ auto schema_json = _resolve_table_schema_json(range).value();
+ auto branch_opt = _resolve_branch(range);
+
+ // 3. Assemble storage options: FE-supplied paimon options + hadoop_conf +
+ // OSS/S3 → AWS_* translations. These feed FileIO only (per
+ // paimon_table_from_schema_json contract); they are NOT merged into the
+ // supplied table schema.
+ auto options = _build_options();
+
+ auto opened_table_key =
+ std::make_tuple(table_path, schema_json, db_name, table_name, branch_opt, options);
+ if (!_handles || !_handles->table || _opened_table_key != opened_table_key) {
+ // A paimon scan reads one table, so the handle is opened at most once per
+ // distinct identity (e.g. re-created after a close); splits of the same
+ // table reuse it and only rebuild the read pipeline below.
+ _close_table();
+ _handles = std::make_unique<PaimonHandles>();
+
+ std::vector<paimon_option> c_options;
+ c_options.reserve(options.size());
+ for (const auto& kv : options) {
+ c_options.push_back(paimon_option {kv.first.c_str(), kv.second.c_str()});
+ }
+
+ LOG(INFO) << "paimon-rust opening table via schema json: db=" << db_name
+ << " table=" << table_name << " path=" << table_path
+ << " branch=" << (branch_opt.has_value() ? branch_opt.value() : "main")
+ << " storage_options=[" << format_options(options) << "]";
+
+ // Build the table directly from the FE-supplied schema JSON. The Rust
+ // side rejects null / empty branch, so we default to paimon's canonical
+ // "main" sentinel when FE did not set paimon_branch (i.e. the table is
+ // on the main branch — matches upstream Identifier.DEFAULT_MAIN_BRANCH).
+ const std::string& branch_str = branch_opt.has_value() ? branch_opt.value() : "main";
+ paimon_result_get_table tbl_res = paimon_table_from_schema_json(
+ table_path.c_str(), schema_json.c_str(), db_name.c_str(), table_name.c_str(),
+ branch_str.c_str(), c_options.empty() ? nullptr : c_options.data(),
+ c_options.size());
+ if (tbl_res.error != nullptr) {
+ return Status::InternalError(
+ "paimon-rust table_from_schema_json failed: db={} table={} err={}", db_name,
+ table_name, consume_error(tbl_res.error));
+ }
+ _handles->table.reset(tbl_res.table);
+ _opened_table_key = std::move(opened_table_key);
+ }
+
+ // 4. Build the read pipeline: read_builder -> case-insensitive -> projection.
+ // Bound the Rust Arrow allocation itself: wide rows must respect the scanner's
+ // adaptive probe size before they are materialized into a Doris block.
+ const auto batch_size = std::to_string(
+ _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+ const paimon_option batch_option {"read.batch-size", batch_size.c_str()};
+ paimon_result_read_builder rb_res =
+ paimon_table_new_read_builder_with_options(_handles->table.get(), &batch_option, 1);
+ if (rb_res.error != nullptr) {
+ return Status::InternalError("paimon-rust new read builder failed: {}",
+ consume_error(rb_res.error));
+ }
+ _handles->read_builder.reset(rb_res.read_builder);
+
+ // Fold column casing on the Rust side so FE-normalized lowercase names
+ // resolve against tables with mixed-case column definitions.
+ if (paimon_error* case_err =
+ paimon_read_builder_with_case_sensitive(_handles->read_builder.get(), false)) {
+ return Status::InternalError("paimon-rust set case_sensitive failed: {}",
+ consume_error(case_err));
+ }
+
+ // Partition keys are excluded: they are materialized from split metadata
+ // (see _fill_non_arrow_columns), and paimon-rust does not emit them.
+ auto read_columns = _build_read_columns();
+ std::vector<const char*> projection;
+ projection.reserve(read_columns.size() + 1);
+ for (const auto& col : read_columns) {
+ projection.push_back(col.c_str());
+ }
+ projection.push_back(nullptr);
+ if (paimon_error* proj_err = paimon_read_builder_with_projection(_handles->read_builder.get(),
+ projection.data())) {
+ return Status::InternalError("paimon-rust set projection failed: {}",
+ consume_error(proj_err));
+ }
+
+ // Convert the scanner conjuncts into a paimon-rust filter and apply it.
+ RETURN_IF_ERROR(_apply_predicate());
+
+ // 5. Deserialize the FE-planned split into a one-split plan, so this
+ // scanner reads exactly the split it was assigned rather than replanning
+ // the whole table. The wire form is identical to what paimon-cpp consumes
+ // (`paimon::table::DataSplit::serialize`).
+ paimon_result_plan plan_res = paimon_plan_from_split_bytes(
+ reinterpret_cast<const uint8_t*>(split_bytes.data()), split_bytes.size());
+ if (plan_res.error != nullptr) {
+ return Status::InternalError("paimon-rust build plan failed: {}",
+ consume_error(plan_res.error));
+ }
+ _handles->plan.reset(plan_res.plan);
+
+ size_t num_splits = paimon_plan_num_splits(_handles->plan.get());
+ if (num_splits == 0) {
+ _split_eof = true;
+ return Status::OK();
+ }
+
+ // 6. Open the arrow stream over the plan.
+ paimon_result_new_read read_res = paimon_read_builder_new_read(_handles->read_builder.get());
+ if (read_res.error != nullptr) {
+ return Status::InternalError("paimon-rust new read failed: {}",
+ consume_error(read_res.error));
+ }
+ _handles->table_read.reset(read_res.read);
+
+ paimon_result_record_batch_reader rdr_res = paimon_table_read_to_arrow(
+ _handles->table_read.get(), _handles->plan.get(), /*offset=*/0, /*length=*/num_splits);
+ if (rdr_res.error != nullptr) {
+ return Status::InternalError("paimon-rust open arrow reader failed: {}",
+ consume_error(rdr_res.error));
+ }
+ _handles->reader.reset(rdr_res.reader);
+ return Status::OK();
+}
+
+void PaimonRustTableReader::_close_split_reader() {
+ _pending_record_batch.reset();
+ _pending_record_batch_offset = 0;
+ if (!_handles) {
+ return;
+ }
+ // Reverse of the declaration order in PaimonHandles.
+ _handles->reader.reset();
+ _handles->table_read.reset();
+ _handles->plan.reset();
+ _handles->read_builder.reset();
+}
+
+void PaimonRustTableReader::_close_table() {
+ if (!_handles) {
+ return;
+ }
+ _close_split_reader();
+ _handles->table.reset();
+ _opened_table_key.reset();
+}
+
+Status PaimonRustTableReader::_apply_predicate() {
+ if (_conjuncts.empty() || !_handles || !_handles->table || !_handles->read_builder) {
+ return Status::OK();
+ }
+ if (_scanner_profile != nullptr) {
+ COUNTER_UPDATE(_rust_predicates_input, _conjuncts.size());
+ for (const auto& conjunct : _conjuncts) {
+ if (conjunct && conjunct->root() && conjunct->root()->is_rf_wrapper()) {
+ COUNTER_UPDATE(_rust_runtime_filters_input, 1);
+ }
+ }
+ }
+ VLOG_DEBUG << "paimon-rust predicate pushdown: " << _conjuncts.size() << " conjunct(s) input";
+ // The conjunct VSlotRefs carry table global indices (positions), so the v2
+ // converter mode resolves fields by the projected column names; partition
+ // keys are excluded because the rust reader does not read them.
+ std::vector<std::string> names;
+ std::vector<DataTypePtr> types;
+ names.reserve(_projected_columns.size());
+ types.reserve(_projected_columns.size());
+ for (const auto& col : _projected_columns) {
+ if (col.is_partition_key) {
+ continue;
+ }
+ names.push_back(col.name);
+ types.push_back(col.type);
+ }
+ PaimonRustPredicateConverter converter(names, types, _handles->table.get());
+ paimon_predicate* predicate = converter.build(_conjuncts);
+ if (_scanner_profile != nullptr) {
+ COUNTER_UPDATE(_rust_predicates_converted, converter.converted_conjuncts());
+ }
+ if (predicate == nullptr) {
+ VLOG_DEBUG << "paimon-rust predicate pushdown: nothing convertible, no filter applied";
+ return Status::OK();
+ }
+ // paimon_read_builder_with_filter consumes the predicate (ownership moves to
+ // the builder) on every path, so we must not free it here.
+ if (paimon_error* err =
+ paimon_read_builder_with_filter(_handles->read_builder.get(), predicate)) {
+ return Status::InternalError("paimon-rust apply filter failed: {}", consume_error(err));
+ }
+ // Count application only after the C API accepts the filter; reader timers
+ // alone cannot distinguish pushdown from Doris residual-only execution.
+ if (_scanner_profile != nullptr) {
+ COUNTER_UPDATE(_rust_predicates_applied, converter.converted_conjuncts());
+ COUNTER_UPDATE(_rust_runtime_filters_applied, converter.converted_runtime_filters());
+ }
+ VLOG_DEBUG << "paimon-rust predicate pushdown: applied";
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_fill_block_from_record_batch(
+ const std::shared_ptr<arrow::RecordBatch>& batch, Block* block, size_t start, size_t rows) {
+ SCOPED_TIMER(_rust_arrow_to_block_time);
+ DORIS_CHECK(batch != nullptr);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK_LE(start + rows, static_cast<size_t>(batch->num_rows()));
+ const auto arrow_start = static_cast<int64_t>(start);
+ const auto arrow_end = static_cast<int64_t>(start + rows);
+ std::unordered_set<size_t> materialized_indices;
+ materialized_indices.reserve(_projected_columns.size());
+ {
+ auto columns_guard = block->mutate_columns_scoped();
+ auto& columns = columns_guard.mutable_columns();
+ for (int c = 0; c < batch->num_columns(); ++c) {
+ const auto& field = batch->schema()->field(c);
+ if (field->name() == VALUE_KIND_FIELD) {
+ continue;
+ }
+ // Projected column names are FE-normalized to lowercase.
+ // paimon-rust's case_sensitive=false setting also case-folds column
+ // names in the schema output, so exact match works — but tolerate
+ // mixed-case Rust output by folding here as well.
+ auto it = _output_name_to_idx.find(field->name());
+ if (it == _output_name_to_idx.end()) {
+ it = _output_name_to_idx.find(to_lower(field->name()));
+ }
+ if (it == _output_name_to_idx.end()) {
+ // Skip columns that are not in the block (e.g. columns dropped by
+ // slot pruning).
+ continue;
+ }
+ const auto output_idx = it->second;
+ if (!materialized_indices.emplace(output_idx).second) {
+ return Status::InternalError("paimon-rust returned duplicate column '{}'",
+ field->name());
+ }
+ try {
+ RETURN_IF_ERROR(columns_guard.get_datatype_by_position(output_idx)
+ ->get_serde()
+ ->read_column_from_arrow(*columns[output_idx],
Review Comment:
[P2] Handle sliced BOOLEAN arrays produced by data evolution. Eligible data-evolution.enabled append-only tables can produce sliced arrays inside Rust's column-wise merge. For overlapping partial-column files whose Parquet row groups have different boundaries (for example source A first emits 17 rows while a BOOLEAN source B emits 32), DataEvolutionReader emits 17 rows and then B.slice(17,15). Arrow 58.4 preserves the BOOLEAN bit offset in ArrayData and the C Data Interface, so this direct SerDe call fails with non-zero array offset is not supported under the default enable_arrow_input_validation=true. Keeping the original imported batch only avoids slices made by this C++ reader; it does not normalize Rust's incoming bit buffers. Normalize these BOOLEAN slices or keep affected data-evolution scans on JNI, and add a differential fixture with mismatched source batch boundaries. See [BOOLEAN export offset](https://github.com/apache/arrow-rs/blob/58.4.0/arrow-array/src/array/boolean_
array.rs#L745-L752).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68825:
URL: https://github.com/apache/doris/pull/68825#discussion_r4230741120
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -575,7 +581,8 @@ public List<ConnectorScanRange> planScan(ConnectorSession session, ConnectorScan
return scanReuse.computeIfAbsent(reuseKey,
key -> Collections.unmodifiableList(planScanInternal(session,
request.getTableHandle(), request.getColumns(), request.getFilter(),
Review Comment:
[P1] Include Rust eligibility in the scan reuse key. The key records only root column names, so with nested pruning enabled two aliases reading the same struct root, one full and one `s.b`, collide even though only the full projection may use Rust. If it plans first, this cache hands its Rust ranges to the nested scan, bypassing the selector's required JNI fallback. The common backend capability set is also absent, so independently selected aliases in a mixed-version BE pool can reuse ranges assigned to an incapable backend. Key the projection shape and backend capabilities before reusing these ranges.
##########
fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRequest.java:
##########
@@ -147,10 +151,20 @@ public boolean isExplainOnly() {
return explainOnly;
}
+ /** Capabilities shared by every backend eligible to execute this scan. */
+ public Set<String> getBackendCapabilities() {
+ return backendCapabilities;
Review Comment:
[P1] Bump the connector SPI major for the new request method. The Paimon plugin now invokes `ConnectorScanRequest.allBackendsSupport`, but the plugin API stays at 12.0. An updated directory plugin can pass the old FE's major-version gate and fail its first scan with `NoSuchMethodError`. Increment the API major and update the pinned surface/version baselines so that older FEs reject the incompatible plugin at load time.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1650,6 +1668,25 @@ private PaimonScanRange buildJniScanRange(Split split, String defaultFileFormat,
return builder.build();
}
+ private PaimonScanRange buildRustScanRange(DataSplit split, FileStoreTable table,
+ PaimonTableHandle handle, String defaultFileFormat,
Review Comment:
[P2] Serialize the table schema once per scan. `buildRustScanRange` runs for every logical split and calls `table.schema()`, strips options, and encodes a fresh JSON string each time; every range retains that identical string. A split-heavy table can consume hundreds of MB or more of FE heap and repeat substantial JSON work during planning. Compute the schema JSON and branch once outside the split loop and share them among range builders.
##########
be/src/format_v2/table/paimon_rust_predicate_converter.cpp:
##########
@@ -0,0 +1,840 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+
+#include <algorithm>
+#include <cctype>
+#include <limits>
+#include <memory>
+#include <utility>
+
+#include "common/logging.h"
+#include "core/column/column_const.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "core/types.h"
+#include "core/value/decimalv2_value.h"
+#include "core/value/timestamptz_value.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vcompound_pred.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vectorized_fn_call.h"
+#include "exprs/vexpr.h"
+#include "exprs/vin_predicate.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+
+namespace doris {
+
+namespace {
+// paimon_datum tags (see paimon.h / bindings/c/src/table.rs::datum_from_c).
+constexpr int32_t kTagBool = 0;
+constexpr int32_t kTagTinyInt = 1;
+constexpr int32_t kTagSmallInt = 2;
+constexpr int32_t kTagInt = 3;
+constexpr int32_t kTagLong = 4;
+constexpr int32_t kTagDouble = 6;
+constexpr int32_t kTagString = 7;
+constexpr int32_t kTagDate = 8;
+constexpr int32_t kTagTimestamp = 10;
+constexpr int32_t kTagDecimal = 12;
+constexpr int32_t kTagBytes = 13;
+
+// paimon decimal precision ceiling (paimon::Decimal::MAX_PRECISION).
+constexpr int32_t kPaimonDecimalMaxPrecision = 38;
+
+// RAII for an owned paimon_predicate*. and/or/not consume their inputs, so we
+// release() before handing pointers to them.
+struct predicate_deleter {
+ void operator()(paimon_predicate* p) const {
+ if (p) {
+ paimon_predicate_free(p);
+ }
+ }
+};
+using predicate_ptr = std::unique_ptr<paimon_predicate, predicate_deleter>;
+
+// RAII for an owned paimon_error*.
+struct error_deleter {
+ void operator()(paimon_error* p) const {
+ if (p) {
+ paimon_error_free(p);
+ }
+ }
+};
+using error_ptr = std::unique_ptr<paimon_error, error_deleter>;
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_predicate_error(paimon_error* err) {
+ error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+} // namespace
+
+PaimonRustPredicateConverter::PaimonRustPredicateConverter(
+ const std::vector<std::string>& column_names, const std::vector<DataTypePtr>& column_types,
+ const paimon_table* table)
+ : _table(table) {
+ DORIS_CHECK(column_names.size() == column_types.size());
+ _columns_by_name.reserve(column_names.size());
+ for (size_t i = 0; i < column_names.size(); ++i) {
+ _columns_by_name.emplace(_normalize_name(column_names[i]),
+ std::make_pair(column_names[i], column_types[i]));
+ }
+ // Paimon TIMESTAMP (wall clock) is stored as epoch-millis-of-the-wall-time
+ // and the DateTimeV2 serde decodes timezone-naive arrow values in UTC, so
+ // timestamp literals convert wall->epoch in UTC. utc_time_zone() needs no
+ // tzdata lookup, so the conversion cannot silently fall back to a
+ // machine-local zone.
+ _utc_tz = cctz::utc_time_zone();
+}
+
+paimon_predicate* PaimonRustPredicateConverter::build(const VExprContextSPtrs& conjuncts) {
+ _converted_conjuncts = 0;
+ _converted_runtime_filters = 0;
+ if (_table == nullptr) {
+ return nullptr;
+ }
+ predicate_ptr result;
+ for (const auto& conjunct : conjuncts) {
+ if (!conjunct || !conjunct->root()) {
+ continue;
+ }
+ auto root = conjunct->root();
+ const bool is_runtime_filter = root->is_rf_wrapper();
+ if (root->is_rf_wrapper()) {
+ if (auto impl = root->get_impl()) {
+ // A null-aware runtime filter (an EQ_FOR_NULL join) must stay
+ // residual: its wrapper execution restores NULL probe rows to
+ // true (RuntimeFilterExpr::change_null_to_true), while the
+ // unwrapped impl — rebuilt as an ordinary IN predicate through
+ // VDirectInPredicate::get_slot_in_expr — treats NULL as
+ // not-in-set and would prune the NULL probes permanently
+ // before the join sees them. Keep the wrapper itself: it fails
+ // every dispatch below, so the conjunct stays in the residual,
+ // and it is safe to execute on selected rows, so later
+ // conjuncts keep pushing. The lance pushdown declines
+ // is_null_aware() filters for the same reason.
+ // is_null_aware() is concrete on RuntimeFilterExpr (the only
+ // class whose is_rf_wrapper() is true), so the dynamic_cast
+ // never fails in practice; the null guard keeps the unwrap for
+ // any future wrapper shape.
+ auto* rf_wrapper = dynamic_cast<RuntimeFilterExpr*>(root.get());
+ if (rf_wrapper == nullptr || !rf_wrapper->is_null_aware()) {
+ root = impl;
+ }
+ }
+ }
+ // Preserve a safe prefix of the conjunct order: a later pushed
+ // predicate (e.g. an arrived IN runtime filter) could otherwise prune
+ // rows on which an earlier error-preserving conjunct —
+ // assert_true(...), a failing cast, ... — must still raise. The v1
+ // partition-pruning path (FileScanner::_init_runtime_filter_partition_
+ // prune_ctxs) stops at is_safe_to_execute_on_selected_rows() for the
+ // same reason, so a convertible predicate after an unsafe conjunct
+ // must not be pushed. Safe conjuncts that cannot be converted keep
+ // the old skip: they cannot raise, so pruning rows before they are
+ // evaluated as the residual never loses an error.
+ if (!_is_safe_to_push(root)) {
+ break;
+ }
+ predicate_ptr pred(_convert_expr(root));
+ if (!pred) {
+ continue;
+ }
+ ++_converted_conjuncts;
+ _converted_runtime_filters += is_runtime_filter;
+ if (!result) {
+ result = std::move(pred);
+ } else {
+ // and consumes both inputs regardless of success.
+ result.reset(paimon_predicate_and(result.release(), pred.release()));
+ if (!result) {
+ _converted_conjuncts = 0;
+ _converted_runtime_filters = 0;
+ return nullptr;
+ }
+ }
+ }
+ return result.release();
+}
+
+bool PaimonRustPredicateConverter::_is_safe_to_push(const VExprSPtr& expr) {
+ if (expr->is_safe_to_execute_on_selected_rows()) {
+ return true;
+ }
+ // VectorizedFnCall::is_safe_to_execute_on_selected_rows() admits a fixed
+ // whitelist that does not include `like`, so a like conjunct would always
+ // stop the safe prefix here and never reach _convert_like. A like call
+ // whose children are themselves safe is just as total as the whitelisted
+ // comparisons: it only compares strings, and its regex is built by escaping
+ // the pattern operand (FunctionLike::convert_like_pattern), so no scanned
+ // value can make it raise — a malformed pattern or escape operand fails
+ // at open() on every execution, regardless of which rows survive. Admit
+ // it here rather than widening the generic whitelist, which gates shared
+ // BE pushdown paths outside this reader's scope.
+ auto* fn = dynamic_cast<VectorizedFnCall*>(expr.get());
+ if (fn == nullptr || _normalize_name(fn->function_name()) != "like") {
+ return false;
+ }
+ for (uint16_t i = 0; i < fn->get_num_children(); ++i) {
+ if (!fn->get_child(i)->is_safe_to_execute_on_selected_rows()) {
+ return false;
+ }
+ }
+ return true;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_expr(const VExprSPtr& expr) {
+ if (!expr) {
+ return nullptr;
+ }
+
+ // Casts are not unwrapped anywhere (predicate root included): a cast node
+ // fails every dispatch below and the conjunct stays in the Doris residual,
+ // mirroring the FE converter, which keeps casted expressions unconverted.
+ if (auto* direct_in = dynamic_cast<VDirectInPredicate*>(expr.get())) {
+ VExprSPtr in_expr;
+ if (direct_in->get_slot_in_expr(in_expr)) {
+ return _convert_in(in_expr);
+ }
+ return nullptr;
+ }
+
+ if (dynamic_cast<VInPredicate*>(expr.get()) != nullptr) {
+ return _convert_in(expr);
+ }
+
+ switch (expr->op()) {
+ case TExprOpcode::COMPOUND_AND:
+ case TExprOpcode::COMPOUND_OR:
+ return _convert_compound(expr);
+ case TExprOpcode::COMPOUND_NOT:
+ return nullptr;
+ case TExprOpcode::EQ:
+ case TExprOpcode::EQ_FOR_NULL:
+ case TExprOpcode::NE:
+ case TExprOpcode::GE:
+ case TExprOpcode::GT:
+ case TExprOpcode::LE:
+ case TExprOpcode::LT:
+ return _convert_binary(expr);
+ default:
+ break;
+ }
+
+ if (auto* fn = dynamic_cast<VectorizedFnCall*>(expr.get())) {
+ auto fn_name = _normalize_name(fn->function_name());
+ if (fn_name == "is_null_pred" || fn_name == "is_not_null_pred") {
+ return _convert_is_null(expr, fn_name);
+ }
+ if (fn_name == "like") {
+ return _convert_like(expr);
+ }
+ }
+
+ return nullptr;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_compound(const VExprSPtr& expr) {
+ if (!expr || expr->get_num_children() != 2) {
+ return nullptr;
+ }
+ predicate_ptr left(_convert_expr(expr->get_child(0)));
+ if (!left) {
+ return nullptr;
+ }
+ predicate_ptr right(_convert_expr(expr->get_child(1)));
+ if (!right) {
+ return nullptr;
+ }
+
+ if (expr->op() == TExprOpcode::COMPOUND_AND) {
+ return paimon_predicate_and(left.release(), right.release());
+ }
+ if (expr->op() == TExprOpcode::COMPOUND_OR) {
+ return paimon_predicate_or(left.release(), right.release());
+ }
+ return nullptr;
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_in(const VExprSPtr& expr) {
+ auto* in_pred = dynamic_cast<VInPredicate*>(expr.get());
+ // Runtime-generated IN sets can exceed VExpr's 16-bit child count. A wrapped
+ // count would push only a prefix and permanently discard valid probe rows.
+ if (!in_pred || expr->children().size() < 2 ||
+ expr->children().size() > std::numeric_limits<uint16_t>::max()) {
+ return nullptr;
+ }
+ auto field_meta = _resolve_field(expr->get_child(0));
+ if (!field_meta) {
+ return nullptr;
+ }
+
+ const auto num_values = expr->get_num_children() - 1;
+ // Reserve up front so the backing strings never reallocate: each datum's
+ // str_data points into storages[i], which must stay stable.
+ std::vector<std::string> storages;
+ std::vector<paimon_datum> datums;
+ storages.reserve(num_values);
+ datums.reserve(num_values);
+ for (uint16_t i = 1; i < expr->get_num_children(); ++i) {
+ // Casted list values are rejected by _convert_literal (the same rule
+ // as the binary RHS): with debug_skip_fold_constant the cast reaches
+ // the BE un-folded, and unwrapping it would filter on the pre-cast
+ // value — in `amount IN (CAST(1.24 AS DECIMAL(10,1)))` Doris keeps
+ // the 1.2 rows while the unwrapped 1.24 push removes them. Rejecting
+ // the value rejects the whole predicate; the residual applies the
+ // cast correctly.
+ auto holder = _convert_literal(expr->get_child(i), field_meta->type);
+ if (!holder) {
+ return nullptr;
+ }
+ storages.emplace_back(std::move(holder->storage));
+ paimon_datum datum = holder->datum;
+ _bind_datum_storage(&datum, storages.back());
+ datums.emplace_back(datum);
+ }
+
+ if (datums.empty()) {
+ return nullptr;
+ }
+ if (in_pred->is_not_in()) {
+ return _take(paimon_predicate_is_not_in(_table, field_meta->column.c_str(), datums.data(),
+ datums.size()));
+ }
+ return _take(paimon_predicate_is_in(_table, field_meta->column.c_str(), datums.data(),
+ datums.size()));
+}
+
+paimon_predicate* PaimonRustPredicateConverter::_convert_binary(const VExprSPtr& expr) {
+ if (!expr || expr->get_num_children() != 2) {
+ return nullptr;
+ }
+ auto field_meta = _resolve_field(expr->get_child(0));
+ if (!field_meta) {
+ return nullptr;
+ }
+ const char* column = field_meta->column.c_str();
+
+ // Convert the RHS first so EQ_FOR_NULL (<=>) only converts when the RHS is
+ // a convertible literal, mirroring the FE converter, which rejects a
+ // non-literal RHS. A column-to-column `a <=> b` must therefore stay in the
+ // Doris residual: it has no single-column rust predicate, and pushing
+ // `a IS NULL` would wrongly discard rows like (1, 1) — rows dropped by the
+ // pushed filter cannot be recovered by the residual conjunct.
+ auto holder = _convert_literal(expr->get_child(1), field_meta->type);
+ if (!holder) {
+ return nullptr;
+ }
+
+ if (expr->op() == TExprOpcode::EQ_FOR_NULL) {
+ return _take(paimon_predicate_is_null(_table, column));
Review Comment:
[P1] Preserve non-null null-safe equality when pushing this predicate. With `disable_nereids_expression_rules='NULL_SAFE_EQUAL_TO_EQUAL'`, `WHERE id <=> 1` reaches this converter with a non-null literal. It currently pushes `id IS NULL`, so Rust removes every matching `id=1` row before Doris applies the residual and the query returns no rows. Push equality for a non-null literal, reserving `IS NULL` for a null literal, or leave the expression as a residual.
##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,924 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn) \
+ struct type##_deleter { \
+ void operator()(paimon_##type* p) const { \
+ if (p) { \
+ freefn(p); \
+ } \
+ } \
+ }; \
+ using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+ explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+ ~ArrowBatch() {
+ auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+ auto* array = static_cast<ArrowArray*>(batch_.array);
+ if (array && array->release) {
+ array->release(array);
+ }
+ if (schema && schema->release) {
+ schema->release(schema);
+ }
+ paimon_arrow_batch_free(batch_);
+ }
+
+ ArrowBatch(const ArrowBatch&) = delete;
+ ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+ ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+ ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+ paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+ error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+ std::string out;
+ for (const auto& kv : options) {
+ if (!out.empty()) {
+ out += ", ";
+ }
+ out += kv.first;
+ }
+ return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+ table_ptr table;
+ read_builder_ptr read_builder;
+ plan_ptr plan;
+ table_read_ptr table_read;
+ record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+ RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+ {
+ // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+ // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.init_timer);
+ // Materialize TIMESTAMP_LTZ in the session timezone — the same
+ // convention as the JNI reader (PaimonJniScanner reads time_zone from
+ // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+ // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+ // of _ctz, so NTZ wall-clock semantics are preserved.
+ DORIS_CHECK(_runtime_state != nullptr);
+ _ctz = _runtime_state->timezone_obj();
+ if (_scanner_profile != nullptr) {
+ file_scan_profile::ensure_hierarchy(_scanner_profile);
+ _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+ file_scan_profile::TABLE_READER);
+ _rust_open_split_time =
+ ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+ _rust_read_batch_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+ _rust_arrow_to_block_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+ _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_converted = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+ }
+ // Projected column name -> fixed output position, registered with both the exact and
+ // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+ // semantics: exact match first, lower-case fallback on lookup).
+ _output_name_to_idx.reserve(_projected_columns.size() * 2);
+ for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+ _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+ _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+ }
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+ // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+ // are idempotent, and clear it only when a new split is explicitly prepared.
+ _close_split_reader();
+ _split_eof = false;
+ _current_range = options.current_range;
+ RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+ if (current_split_pruned()) {
+ return Status::OK();
+ }
+ if (_is_table_level_count_active()) {
+ // No rust pipeline is opened; get_block emits the synthetic count rows.
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.prepare_split_timer);
+ // Open may dominate the scan or fail before get_block; include it in the parent timer.
+ SCOPED_TIMER(_rust_total_time);
+ SCOPED_TIMER(_rust_open_split_time);
+ RETURN_IF_ERROR(_open_split_reader(options.current_range));
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.exec_timer);
+ SCOPED_TIMER(_rust_total_time);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK(eos != nullptr);
+ DORIS_CHECK(block->columns() == _projected_columns.size());
+ block->clear_column_data(_projected_columns.size());
+ *eos = false;
+
+ if (_is_table_level_count_active()) {
+ return _read_table_level_count(block, eos);
+ }
+
+ // num_splits == 0 yields an empty (but valid) stream: report EOF.
+ if (_split_eof) {
+ *eos = true;
+ return Status::OK();
+ }
+ if (!_handles || !_handles->reader) {
+ return Status::InternalError("paimon-rust reader is not initialized");
+ }
+
+ while (true) {
+ // Mirror the base TableReader cancellation contract so a cancelled query does not
+ // drain the whole split.
+ if (_io_ctx != nullptr && _io_ctx->should_stop) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ paimon_result_next_batch next;
+ {
+ SCOPED_TIMER(_rust_read_batch_time);
+ next = paimon_record_batch_reader_next(_handles->reader.get());
+ }
+ if (next.error != nullptr) {
+ return Status::InternalError("paimon-rust read batch failed: {}",
+ consume_error(next.error));
+ }
+ // End of stream: both pointers are null.
+ if (next.batch.array == nullptr && next.batch.schema == nullptr) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ // RAII: the batch's Arrow release callbacks + container free run when
+ // `batch` leaves this scope, including on any early return.
+ ArrowBatch batch(next.batch);
+
+ auto* c_array = batch.array();
+ auto* c_schema = batch.schema();
+ arrow::Result<std::shared_ptr<arrow::RecordBatch>> import_result =
+ arrow::ImportRecordBatch(c_array, c_schema);
+ if (!import_result.ok()) {
+ return Status::InternalError("failed to import paimon-rust arrow batch: {}",
+ import_result.status().message());
+ }
+
+ auto record_batch = std::move(import_result).ValueUnsafe();
+ const auto rows = static_cast<size_t>(record_batch->num_rows());
+ if (rows == 0) {
+ // Skip empty batches and keep draining the stream.
+ continue;
+ }
+ RETURN_IF_ERROR(_fill_block_from_record_batch(record_batch, block, rows));
+ _record_scan_rows(rows);
+ *eos = false;
+ return Status::OK();
+ }
+}
+
+Status PaimonRustTableReader::abort_split() {
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.close_timer);
+ _close_split_reader();
+ _split_eof = false;
+ }
+ return format::TableReader::abort_split();
+}
+
+#ifdef BE_TEST
+std::string PaimonRustTableReader::TEST_format_options(
+ const std::map<std::string, std::string>& options) {
+ return format_options(options);
+}
+
+std::map<std::string, std::string> PaimonRustTableReader::TEST_build_options(
+ TFileScanRangeParams* scan_params, const TFileRangeDesc& range) {
+ TFileScanRangeParams* previous_params = _scan_params;
+ TFileRangeDesc previous_range = _current_range;
+ _scan_params = scan_params;
+ _current_range = range;
+ std::map<std::string, std::string> options = _build_options();
+ _scan_params = previous_params;
+ _current_range = std::move(previous_range);
+ return options;
+}
+#endif
+
+Status PaimonRustTableReader::close() {
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.close_timer);
+ _close_split_reader();
+ _close_table();
+ }
+ return format::TableReader::close();
+}
+
+Status PaimonRustTableReader::_validate_rust_split(const TFileRangeDesc& range) const {
+ if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) {
+ return Status::InternalError(
+ "missing paimon_params for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ const auto& params = range.table_format_params.paimon_params;
+ if (!params.__isset.paimon_split || params.paimon_split.empty()) {
+ return Status::InternalError(
+ "missing paimon_split for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ if (params.__isset.reader_type && params.reader_type != TPaimonReaderType::PAIMON_RUST) {
+ return Status::InternalError(
+ "invalid reader_type for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ if (!_resolve_table_path(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing paimon_table; cannot resolve paimon table location");
+ }
+ if (!_resolve_db_name(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing db_name; cannot open paimon table via schema json");
+ }
+ if (!_resolve_table_name(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing table_name; cannot open paimon table via schema json");
+ }
+ if (!_resolve_table_schema_json(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing paimon_table_schema_json; cannot open paimon table via "
+ "schema json");
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_open_split_reader(const TFileRangeDesc& range) {
+ // 1. Decode the FE-planned split first so we fail fast (and without any
+ // filesystem IO) when it is missing or malformed.
+ std::string split_bytes;
+ RETURN_IF_ERROR(_decode_split_bytes(&split_bytes));
+
+ // 2. Resolve identifier + table_path + FE-supplied TableSchema JSON.
+ auto table_path = _resolve_table_path(range).value();
+ auto db_name = _resolve_db_name(range).value();
+ auto table_name = _resolve_table_name(range).value();
+ auto schema_json = _resolve_table_schema_json(range).value();
+ auto branch_opt = _resolve_branch(range);
+
+ // 3. Assemble storage options: FE-supplied paimon options + hadoop_conf +
+ // OSS/S3 → AWS_* translations. These feed FileIO only (per
+ // paimon_table_from_schema_json contract); they are NOT merged into the
+ // supplied table schema.
+ auto options = _build_options();
+
+ auto opened_table_key =
+ std::make_tuple(table_path, schema_json, db_name, table_name, branch_opt, options);
+ if (!_handles || !_handles->table || _opened_table_key != opened_table_key) {
+ // A paimon scan reads one table, so the handle is opened at most once per
+ // distinct identity (e.g. re-created after a close); splits of the same
+ // table reuse it and only rebuild the read pipeline below.
+ _close_table();
+ _handles = std::make_unique<PaimonHandles>();
+
+ std::vector<paimon_option> c_options;
+ c_options.reserve(options.size());
+ for (const auto& kv : options) {
+ c_options.push_back(paimon_option {kv.first.c_str(), kv.second.c_str()});
+ }
+
+ LOG(INFO) << "paimon-rust opening table via schema json: db=" << db_name
+ << " table=" << table_name << " path=" << table_path
+ << " branch=" << (branch_opt.has_value() ? branch_opt.value() : "main")
+ << " storage_options=[" << format_options(options) << "]";
+
+ // Build the table directly from the FE-supplied schema JSON. The Rust
+ // side rejects null / empty branch, so we default to paimon's canonical
+ // "main" sentinel when FE did not set paimon_branch (i.e. the table is
+ // on the main branch — matches upstream Identifier.DEFAULT_MAIN_BRANCH).
+ const std::string& branch_str = branch_opt.has_value() ? branch_opt.value() : "main";
+ paimon_result_get_table tbl_res = paimon_table_from_schema_json(
+ table_path.c_str(), schema_json.c_str(), db_name.c_str(), table_name.c_str(),
+ branch_str.c_str(), c_options.empty() ? nullptr : c_options.data(),
+ c_options.size());
+ if (tbl_res.error != nullptr) {
+ return Status::InternalError(
+ "paimon-rust table_from_schema_json failed: db={} table={} err={}", db_name,
+ table_name, consume_error(tbl_res.error));
+ }
+ _handles->table.reset(tbl_res.table);
+ _opened_table_key = std::move(opened_table_key);
+ }
+
+ // 4. Build the read pipeline: read_builder -> case-insensitive -> projection.
+ // Bound the Rust Arrow allocation itself: wide rows must respect the scanner's
+ // adaptive probe size before they are materialized into a Doris block.
+ const auto batch_size = std::to_string(
+ _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+ const paimon_option batch_option {"read.batch-size", batch_size.c_str()};
+ paimon_result_read_builder rb_res =
+ paimon_table_new_read_builder_with_options(_handles->table.get(), &batch_option, 1);
+ if (rb_res.error != nullptr) {
+ return Status::InternalError("paimon-rust new read builder failed: {}",
+ consume_error(rb_res.error));
+ }
+ _handles->read_builder.reset(rb_res.read_builder);
+
+ // Fold column casing on the Rust side so FE-normalized lowercase names
+ // resolve against tables with mixed-case column definitions.
+ if (paimon_error* case_err =
+ paimon_read_builder_with_case_sensitive(_handles->read_builder.get(), false)) {
+ return Status::InternalError("paimon-rust set case_sensitive failed: {}",
+ consume_error(case_err));
+ }
+
+ // Partition keys are excluded: they are materialized from split metadata
+ // (see _fill_non_arrow_columns), and paimon-rust does not emit them.
+ auto read_columns = _build_read_columns();
+ std::vector<const char*> projection;
+ projection.reserve(read_columns.size() + 1);
+ for (const auto& col : read_columns) {
+ projection.push_back(col.c_str());
+ }
+ projection.push_back(nullptr);
+ if (paimon_error* proj_err = paimon_read_builder_with_projection(_handles->read_builder.get(),
+ projection.data())) {
+ return Status::InternalError("paimon-rust set projection failed: {}",
+ consume_error(proj_err));
+ }
+
+ // Convert the scanner conjuncts into a paimon-rust filter and apply it.
+ RETURN_IF_ERROR(_apply_predicate());
+
+ // 5. Deserialize the FE-planned split into a one-split plan, so this
+ // scanner reads exactly the split it was assigned rather than replanning
+ // the whole table. The wire form is identical to what paimon-cpp consumes
+ // (`paimon::table::DataSplit::serialize`).
+ paimon_result_plan plan_res = paimon_plan_from_split_bytes(
+ reinterpret_cast<const uint8_t*>(split_bytes.data()), split_bytes.size());
+ if (plan_res.error != nullptr) {
+ return Status::InternalError("paimon-rust build plan failed: {}",
+ consume_error(plan_res.error));
+ }
+ _handles->plan.reset(plan_res.plan);
+
+ size_t num_splits = paimon_plan_num_splits(_handles->plan.get());
+ if (num_splits == 0) {
+ _split_eof = true;
+ return Status::OK();
+ }
+
+ // 6. Open the arrow stream over the plan.
+ paimon_result_new_read read_res = paimon_read_builder_new_read(_handles->read_builder.get());
+ if (read_res.error != nullptr) {
+ return Status::InternalError("paimon-rust new read failed: {}",
+ consume_error(read_res.error));
+ }
+ _handles->table_read.reset(read_res.read);
+
+ paimon_result_record_batch_reader rdr_res = paimon_table_read_to_arrow(
+ _handles->table_read.get(), _handles->plan.get(), /*offset=*/0, /*length=*/num_splits);
+ if (rdr_res.error != nullptr) {
+ return Status::InternalError("paimon-rust open arrow reader failed: {}",
+ consume_error(rdr_res.error));
+ }
+ _handles->reader.reset(rdr_res.reader);
+ return Status::OK();
+}
+
+void PaimonRustTableReader::_close_split_reader() {
+ if (!_handles) {
+ return;
+ }
+ // Reverse of the declaration order in PaimonHandles.
+ _handles->reader.reset();
+ _handles->table_read.reset();
+ _handles->plan.reset();
+ _handles->read_builder.reset();
+}
+
+void PaimonRustTableReader::_close_table() {
+ if (!_handles) {
+ return;
+ }
+ _close_split_reader();
+ _handles->table.reset();
+ _opened_table_key.reset();
+}
+
+Status PaimonRustTableReader::_apply_predicate() {
+ if (_conjuncts.empty() || !_handles || !_handles->table || !_handles->read_builder) {
+ return Status::OK();
+ }
+ if (_scanner_profile != nullptr) {
+ COUNTER_UPDATE(_rust_predicates_input, _conjuncts.size());
+ for (const auto& conjunct : _conjuncts) {
+ if (conjunct && conjunct->root() && conjunct->root()->is_rf_wrapper()) {
+ COUNTER_UPDATE(_rust_runtime_filters_input, 1);
+ }
+ }
+ }
+ LOG(INFO) << "paimon-rust predicate pushdown: " << _conjuncts.size() << " conjunct(s) input";
Review Comment:
[P2] Avoid INFO logging for every filtered split. `_apply_predicate` runs once per Rust logical split and emits this line plus an applied/none line whenever conjuncts exist. A 50,000-split scan therefore writes about 100,000 INFO records, adding avoidable log I/O and rotation pressure to a hot scan path. Put per-split details behind verbose logging or use the existing predicate profile counters for the aggregate result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68825:
URL: https://github.com/apache/doris/pull/68825#discussion_r4237034237
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -398,6 +399,11 @@ struct TPaimonFileDesc {
// Original Paimon RawFile.path() before Doris storage path normalization. Native readers use this
// to materialize the public file-location metadata column.
18: optional string original_file_path;
+ // paimon-rust: TableSchema serialized via JsonSerdeUtil.toJson(table.schema()).
+ // Consumed by paimon_table_from_schema_json on BE; skips catalog/warehouse resolution.
+ 19: optional string paimon_table_schema_json;
Review Comment:
[P1] Keep Paimon Rust payload IDs compatible across release branches. Branches 4.1 and 4.2 use field 18 for schema JSON and field 19 for branch, whereas this new field 19 is schema JSON and field 20 is branch. Those branches already advertise Rust capability, so a mixed-version FE/BE pair can select Rust and decode the schema as a branch, or find no schema at all, causing scans to fail. Master already used field 18 for `original_file_path` before this PR, so a compatible versioned payload/capability or equivalent migration is needed; merely renumbering that existing field would break master-side compatibility.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -991,16 +1009,24 @@ && hasTrustworthyLimitAccounting(table)
dataSplit.bucket()));
}
} else {
- // JNI reader path
+ // Logical reader path: use paimon-rust only for the compatibility-checked subset.
if (ignoreJni) {
// FIX-L14: ignore_split_type=IGNORE_JNI drops JNI splits (legacy getSplits:483).
continue;
}
if (requiresMetadataColumns) {
validateMetadataColumnReader(true, false);
}
- ranges.add(buildJniScanRange(dataSplit, defaultFileFormat,
- partitionValues, true, weightDenominator));
+ if (rustReaderSelector != null && rustReaderSelector.canRead(dataSplit)) {
Review Comment:
[P2] Preserve the `force_jni_scanner` escape hatch when Rust is enabled. That session variable is documented as forcing JNI for external tables. It disables the native arm, but this logical arm still selects Rust when `enable_paimon_rust_reader=true` and the split is eligible; the added differential test uses exactly that combination. A user setting the force flag to bypass a failing Rust read will still run Rust. Make the override reach the Rust selector, and use a separate logical-split test mechanism if needed.
##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,924 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn) \
+ struct type##_deleter { \
+ void operator()(paimon_##type* p) const { \
+ if (p) { \
+ freefn(p); \
+ } \
+ } \
+ }; \
+ using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+ explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+ ~ArrowBatch() {
+ auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+ auto* array = static_cast<ArrowArray*>(batch_.array);
+ if (array && array->release) {
+ array->release(array);
+ }
+ if (schema && schema->release) {
+ schema->release(schema);
+ }
+ paimon_arrow_batch_free(batch_);
+ }
+
+ ArrowBatch(const ArrowBatch&) = delete;
+ ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+ ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+ ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+ paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+ error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+ std::string out;
+ for (const auto& kv : options) {
+ if (!out.empty()) {
+ out += ", ";
+ }
+ out += kv.first;
+ }
+ return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+ table_ptr table;
+ read_builder_ptr read_builder;
+ plan_ptr plan;
+ table_read_ptr table_read;
+ record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+ RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+ {
+ // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+ // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.init_timer);
+ // Materialize TIMESTAMP_LTZ in the session timezone — the same
+ // convention as the JNI reader (PaimonJniScanner reads time_zone from
+ // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+ // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+ // of _ctz, so NTZ wall-clock semantics are preserved.
+ DORIS_CHECK(_runtime_state != nullptr);
+ _ctz = _runtime_state->timezone_obj();
+ if (_scanner_profile != nullptr) {
+ file_scan_profile::ensure_hierarchy(_scanner_profile);
+ _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+ file_scan_profile::TABLE_READER);
+ _rust_open_split_time =
+ ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+ _rust_read_batch_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+ _rust_arrow_to_block_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+ _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_converted = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+ }
+ // Projected column name -> fixed output position, registered with both the exact and
+ // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+ // semantics: exact match first, lower-case fallback on lookup).
+ _output_name_to_idx.reserve(_projected_columns.size() * 2);
+ for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+ _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+ _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+ }
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+ // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+ // are idempotent, and clear it only when a new split is explicitly prepared.
+ _close_split_reader();
+ _split_eof = false;
+ _current_range = options.current_range;
+ RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+ if (current_split_pruned()) {
+ return Status::OK();
+ }
+ if (_is_table_level_count_active()) {
+ // No rust pipeline is opened; get_block emits the synthetic count rows.
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.prepare_split_timer);
+ // Open may dominate the scan or fail before get_block; include it in the parent timer.
+ SCOPED_TIMER(_rust_total_time);
+ SCOPED_TIMER(_rust_open_split_time);
+ RETURN_IF_ERROR(_open_split_reader(options.current_range));
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.exec_timer);
+ SCOPED_TIMER(_rust_total_time);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK(eos != nullptr);
+ DORIS_CHECK(block->columns() == _projected_columns.size());
+ block->clear_column_data(_projected_columns.size());
+ *eos = false;
+
+ if (_is_table_level_count_active()) {
+ return _read_table_level_count(block, eos);
+ }
+
+ // num_splits == 0 yields an empty (but valid) stream: report EOF.
+ if (_split_eof) {
+ *eos = true;
+ return Status::OK();
+ }
+ if (!_handles || !_handles->reader) {
+ return Status::InternalError("paimon-rust reader is not initialized");
+ }
+
+ while (true) {
+ // Mirror the base TableReader cancellation contract so a cancelled query does not
+ // drain the whole split.
+ if (_io_ctx != nullptr && _io_ctx->should_stop) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ paimon_result_next_batch next;
+ {
+ SCOPED_TIMER(_rust_read_batch_time);
+ next = paimon_record_batch_reader_next(_handles->reader.get());
+ }
+ if (next.error != nullptr) {
+ return Status::InternalError("paimon-rust read batch failed: {}",
+ consume_error(next.error));
+ }
+ // End of stream: both pointers are null.
+ if (next.batch.array == nullptr && next.batch.schema == nullptr) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ // RAII: the batch's Arrow release callbacks + container free run when
+ // `batch` leaves this scope, including on any early return.
+ ArrowBatch batch(next.batch);
+
+ auto* c_array = batch.array();
+ auto* c_schema = batch.schema();
+ arrow::Result<std::shared_ptr<arrow::RecordBatch>> import_result =
+ arrow::ImportRecordBatch(c_array, c_schema);
+ if (!import_result.ok()) {
+ return Status::InternalError("failed to import paimon-rust arrow batch: {}",
+ import_result.status().message());
+ }
+
+ auto record_batch = std::move(import_result).ValueUnsafe();
+ const auto rows = static_cast<size_t>(record_batch->num_rows());
+ if (rows == 0) {
+ // Skip empty batches and keep draining the stream.
+ continue;
+ }
+ RETURN_IF_ERROR(_fill_block_from_record_batch(record_batch, block, rows));
+ _record_scan_rows(rows);
+ *eos = false;
+ return Status::OK();
+ }
+}
+
+Status PaimonRustTableReader::abort_split() {
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.close_timer);
+ _close_split_reader();
+ _split_eof = false;
+ }
+ return format::TableReader::abort_split();
+}
+
+#ifdef BE_TEST
+std::string PaimonRustTableReader::TEST_format_options(
+ const std::map<std::string, std::string>& options) {
+ return format_options(options);
+}
+
+std::map<std::string, std::string> PaimonRustTableReader::TEST_build_options(
+ TFileScanRangeParams* scan_params, const TFileRangeDesc& range) {
+ TFileScanRangeParams* previous_params = _scan_params;
+ TFileRangeDesc previous_range = _current_range;
+ _scan_params = scan_params;
+ _current_range = range;
+ std::map<std::string, std::string> options = _build_options();
+ _scan_params = previous_params;
+ _current_range = std::move(previous_range);
+ return options;
+}
+#endif
+
+Status PaimonRustTableReader::close() {
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.close_timer);
+ _close_split_reader();
+ _close_table();
+ }
+ return format::TableReader::close();
+}
+
+Status PaimonRustTableReader::_validate_rust_split(const TFileRangeDesc& range) const {
+ if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) {
+ return Status::InternalError(
+ "missing paimon_params for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ const auto& params = range.table_format_params.paimon_params;
+ if (!params.__isset.paimon_split || params.paimon_split.empty()) {
+ return Status::InternalError(
+ "missing paimon_split for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ if (params.__isset.reader_type && params.reader_type != TPaimonReaderType::PAIMON_RUST) {
+ return Status::InternalError(
+ "invalid reader_type for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ if (!_resolve_table_path(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing paimon_table; cannot resolve paimon table location");
+ }
+ if (!_resolve_db_name(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing db_name; cannot open paimon table via schema json");
+ }
+ if (!_resolve_table_name(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing table_name; cannot open paimon table via schema json");
+ }
+ if (!_resolve_table_schema_json(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing paimon_table_schema_json; cannot open paimon table via "
+ "schema json");
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_open_split_reader(const TFileRangeDesc& range) {
+ // 1. Decode the FE-planned split first so we fail fast (and without any
+ // filesystem IO) when it is missing or malformed.
+ std::string split_bytes;
+ RETURN_IF_ERROR(_decode_split_bytes(&split_bytes));
+
+ // 2. Resolve identifier + table_path + FE-supplied TableSchema JSON.
+ auto table_path = _resolve_table_path(range).value();
+ auto db_name = _resolve_db_name(range).value();
+ auto table_name = _resolve_table_name(range).value();
+ auto schema_json = _resolve_table_schema_json(range).value();
+ auto branch_opt = _resolve_branch(range);
+
+ // 3. Assemble storage options: FE-supplied paimon options + hadoop_conf +
+ // OSS/S3 → AWS_* translations. These feed FileIO only (per
+ // paimon_table_from_schema_json contract); they are NOT merged into the
+ // supplied table schema.
+ auto options = _build_options();
+
+ auto opened_table_key =
+ std::make_tuple(table_path, schema_json, db_name, table_name, branch_opt, options);
+ if (!_handles || !_handles->table || _opened_table_key != opened_table_key) {
+ // A paimon scan reads one table, so the handle is opened at most once per
+ // distinct identity (e.g. re-created after a close); splits of the same
+ // table reuse it and only rebuild the read pipeline below.
+ _close_table();
+ _handles = std::make_unique<PaimonHandles>();
+
+ std::vector<paimon_option> c_options;
+ c_options.reserve(options.size());
+ for (const auto& kv : options) {
+ c_options.push_back(paimon_option {kv.first.c_str(), kv.second.c_str()});
+ }
+
+ LOG(INFO) << "paimon-rust opening table via schema json: db=" << db_name
+ << " table=" << table_name << " path=" << table_path
+ << " branch=" << (branch_opt.has_value() ? branch_opt.value() : "main")
+ << " storage_options=[" << format_options(options) << "]";
+
+ // Build the table directly from the FE-supplied schema JSON. The Rust
+ // side rejects null / empty branch, so we default to paimon's canonical
+ // "main" sentinel when FE did not set paimon_branch (i.e. the table is
+ // on the main branch — matches upstream Identifier.DEFAULT_MAIN_BRANCH).
+ const std::string& branch_str = branch_opt.has_value() ? branch_opt.value() : "main";
+ paimon_result_get_table tbl_res = paimon_table_from_schema_json(
+ table_path.c_str(), schema_json.c_str(), db_name.c_str(), table_name.c_str(),
+ branch_str.c_str(), c_options.empty() ? nullptr : c_options.data(),
+ c_options.size());
+ if (tbl_res.error != nullptr) {
+ return Status::InternalError(
+ "paimon-rust table_from_schema_json failed: db={} table={} err={}", db_name,
+ table_name, consume_error(tbl_res.error));
+ }
+ _handles->table.reset(tbl_res.table);
+ _opened_table_key = std::move(opened_table_key);
+ }
+
+ // 4. Build the read pipeline: read_builder -> case-insensitive -> projection.
+ // Bound the Rust Arrow allocation itself: wide rows must respect the scanner's
+ // adaptive probe size before they are materialized into a Doris block.
+ const auto batch_size = std::to_string(
+ _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+ const paimon_option batch_option {"read.batch-size", batch_size.c_str()};
Review Comment:
[P2] Apply adaptive row caps to the active Rust stream. FileScannerV2 seeds 32 rows before `prepare_split`, then learns bytes per row and calls `set_batch_size` before later `get_block` calls. Here `read.batch-size` is fixed when the Rust builder opens, while the inherited setter only changes `_batch_size`. A split with 1 MiB rows therefore keeps emitting roughly 32 MiB Arrow batches after the scanner asks for about eight rows to meet its default 8 MiB target. The added test changes the cap only between splits; please cover and honor a cap change after the first batch.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonRustReaderCapabilities.java:
##########
@@ -0,0 +1,281 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
+
+import com.google.common.collect.ImmutableSet;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeChecks;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.DecimalType;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.types.VarCharType;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/** Compatibility checks for the pinned paimon-rust reader, beyond storage capabilities. */
+final class PaimonRustReaderCapabilities {
+ private static final Set<String> SUPPORTED_AGGREGATE_NAMES = ImmutableSet.of(
+ "sum", "product", "min", "max", "last_value", "first_value", "last_non_null_value",
+ "first_non_null_value", "first_not_null_value", "bool_and", "bool_or", "listagg");
+ private final FileStoreTable table;
+ private final TableSchema schema;
+ private final boolean tableCompatible;
+ private final Map<Long, Boolean> compatibleFileSchemas = new ConcurrentHashMap<>();
+
+ PaimonRustReaderCapabilities(FileStoreTable table, List<ConnectorColumnHandle> columns) {
+ this.table = table;
+ this.schema = table.schema();
+ this.tableCompatible = schema != null && hasCompatibleSchemaVersion(schema)
+ && hasFullNestedProjection(columns) && hasCompatibleMergeEngine(schema)
+ && hasCompatibleAggregates(schema);
+ }
+
+ boolean canRead(DataSplit split) {
+ if (!tableCompatible) {
+ return false;
+ }
+ // The pinned merge reader retains losing input batches until an output batch fills.
+ // Neither zero deletes nor a small read.batch-size bounds this across multiple files.
+ if (!schema.primaryKeys().isEmpty() && split.dataFiles().size() > 1) {
+ return false;
+ }
+ for (DataFileMeta file : split.dataFiles()) {
+ // Sort-merge retains consumed batches until it emits enough rows. Retracts can
+ // produce an unbounded zero-output prefix; aggregation also rejects retracts.
+ // Unknown counts must stay on JNI, including old files without this statistic.
+ if (!schema.primaryKeys().isEmpty() && file.deleteRowCount().orElse(-1L) != 0L) {
+ return false;
+ }
+ if (file.schemaId() != schema.id() && !compatibleFileSchemas.computeIfAbsent(
Review Comment:
[P1] Guard current-schema nanosecond timestamps before Rust selection. Paimon permits `TIMESTAMP(9)` year-3000 values and stores precision >6 as Parquet INT96 ([type range](https://paimon.apache.org/docs/master/concepts/data-types/); [Parquet mapping](https://paimon.apache.org/docs/master/concepts/spec/fileformat/)). The pinned Rust/Parquet reader maps INT96 to signed i64 Arrow nanoseconds, and [`Int96::to_nanos` wraps on overflow](https://github.com/apache/arrow-rs/blob/58.4.0/parquet/src/data_type.rs#L110-L119) beyond year 2262. This `schemaId() != schema.id()` condition skips the only timestamp precision check for current-schema files, so an eligible logical split can return a different date from JNI without error. Gate precision 7–9, including nested leaves, until decoding preserves the full range; add a year-3000 parity case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6185258505
# BE UT Coverage Report
Increment line coverage `100.00% (31/31)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/b3e30fa9240264b19d185ee444c6531d4106a8ff_b3e30fa9240264b19d185ee444c6531d4106a8ff/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/b3e30fa9240264b19d185ee444c6531d4106a8ff_b3e30fa9240264b19d185ee444c6531d4106a8ff/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 66.18% (31645/47817) |
| Line Coverage | 50.99% (334292/655632) |
| Region Coverage | 46.30% (270121/583450) |
| Branch Coverage | 47.91% (126796/264643) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6187819730
# FE Regression Coverage Report
Increment line coverage ` 20.00% (19/95)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6095464254
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6095738470
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/701da299bb5dc73be36da3cf7e3346905c1250bd_701da299bb5dc73be36da3cf7e3346905c1250bd_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/701da299bb5dc73be36da3cf7e3346905c1250bd_701da299bb5dc73be36da3cf7e3346905c1250bd_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.84% (2079/2671) |
| Line Coverage | 66.04% (38197/57840) |
| Region Coverage | 53.49% (35798/66920) |
| Branch Coverage | 56.95% (11517/20223) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6161889666
# BE UT Coverage Report
Increment line coverage `100.00% (31/31)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/13e491640aa8d4bb03224c0190f1c40ec26b9db1_13e491640aa8d4bb03224c0190f1c40ec26b9db1/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/13e491640aa8d4bb03224c0190f1c40ec26b9db1_13e491640aa8d4bb03224c0190f1c40ec26b9db1/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 65.91% (31469/47746) |
| Line Coverage | 50.65% (331626/654805) |
| Region Coverage | 46.02% (268095/582553) |
| Branch Coverage | 47.68% (125945/264159) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6168561428
# BE UT Coverage Report
Increment line coverage `100.00% (31/31)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/8d52953c3037b078816cdc14d5695e371b5cfacf_8d52953c3037b078816cdc14d5695e371b5cfacf/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/8d52953c3037b078816cdc14d5695e371b5cfacf_8d52953c3037b078816cdc14d5695e371b5cfacf/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 66.16% (31631/47809) |
| Line Coverage | 50.97% (334128/655519) |
| Region Coverage | 46.29% (270014/583346) |
| Branch Coverage | 47.90% (126753/264600) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on code in PR #68825:
URL: https://github.com/apache/doris/pull/68825#discussion_r4278368880
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -897,6 +911,16 @@ && hasTrustworthyLimitAccounting(table)
}
}
+ boolean incrementalRead = pinnedOptions.containsKey("incremental-between")
+ || pinnedOptions.containsKey("incremental-between-timestamp");
+ PaimonRustReaderSelector rustReaderSelector = table instanceof FileStoreTable
+ && !requiresMetadataColumns
+ ? new PaimonRustReaderSelector(session, backendsSupportRust,
+ (FileStoreTable) table, columns, backendStorageProperties(),
Review Comment:
Thanks. The original #66227 selector likewise derives Rust eligibility and transport from typed backend storage properties. The raw-option precedence gap is valid, but changing the effective-option transport in this forward-port would diverge from the picked implementation. We will address the effective Paimon FileIO configuration in a focused follow-up and keep this PR aligned with the source. Resolving as deferred, not fixed.
##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn) \
+ struct type##_deleter { \
+ void operator()(paimon_##type* p) const { \
+ if (p) { \
+ freefn(p); \
+ } \
+ } \
+ }; \
+ using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+ explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+ ~ArrowBatch() {
+ auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+ auto* array = static_cast<ArrowArray*>(batch_.array);
+ if (array && array->release) {
+ array->release(array);
+ }
+ if (schema && schema->release) {
+ schema->release(schema);
+ }
+ paimon_arrow_batch_free(batch_);
+ }
+
+ ArrowBatch(const ArrowBatch&) = delete;
+ ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+ ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+ ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+ paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+ rust_handles::error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+ std::string out;
+ for (const auto& kv : options) {
+ if (!out.empty()) {
+ out += ", ";
+ }
+ out += kv.first;
+ }
+ return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+ rust_handles::table_ptr table;
+ rust_handles::read_builder_ptr read_builder;
+ rust_handles::plan_ptr plan;
+ rust_handles::table_read_ptr table_read;
+ rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+ RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+ {
+ // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+ // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.init_timer);
+ // Materialize TIMESTAMP_LTZ in the session timezone — the same
+ // convention as the JNI reader (PaimonJniScanner reads time_zone from
+ // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+ // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+ // of _ctz, so NTZ wall-clock semantics are preserved.
+ DORIS_CHECK(_runtime_state != nullptr);
+ _ctz = _runtime_state->timezone_obj();
+ if (_scanner_profile != nullptr) {
+ file_scan_profile::ensure_hierarchy(_scanner_profile);
+ _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+ file_scan_profile::TABLE_READER);
+ _rust_open_split_time =
+ ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+ _rust_read_batch_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+ _rust_arrow_to_block_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+ _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_converted = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+ }
+ // Projected column name -> fixed output position, registered with both the exact and
+ // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+ // semantics: exact match first, lower-case fallback on lookup).
+ _output_name_to_idx.reserve(_projected_columns.size() * 2);
+ for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+ _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+ _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+ }
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+ // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+ // are idempotent, and clear it only when a new split is explicitly prepared.
+ _close_split_reader();
+ _split_eof = false;
+ _current_range = options.current_range;
+ RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+ if (current_split_pruned()) {
+ return Status::OK();
+ }
+ if (_is_table_level_count_active()) {
+ // No rust pipeline is opened; get_block emits the synthetic count rows.
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.prepare_split_timer);
+ // Open may dominate the scan or fail before get_block; include it in the parent timer.
+ SCOPED_TIMER(_rust_total_time);
+ SCOPED_TIMER(_rust_open_split_time);
+ RETURN_IF_ERROR(_open_split_reader(options.current_range));
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.exec_timer);
+ SCOPED_TIMER(_rust_total_time);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK(eos != nullptr);
+ DORIS_CHECK(block->columns() == _projected_columns.size());
+ block->clear_column_data(_projected_columns.size());
+ *eos = false;
+
+ if (_is_table_level_count_active()) {
+ return _read_table_level_count(block, eos);
+ }
+
+ // num_splits == 0 yields an empty (but valid) stream: report EOF.
+ if (_split_eof) {
+ *eos = true;
+ return Status::OK();
+ }
+ if (!_handles || !_handles->reader) {
+ return Status::InternalError("paimon-rust reader is not initialized");
+ }
+
+ while (true) {
+ // Mirror the base TableReader cancellation contract so a cancelled query does not
+ // drain the whole split.
+ if (_io_ctx != nullptr && _io_ctx->should_stop) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ if (!_pending_record_batch) {
+ paimon_result_next_batch next;
+ {
+ SCOPED_TIMER(_rust_read_batch_time);
+ next = paimon_record_batch_reader_next(_handles->reader.get());
+ }
+ if (next.error != nullptr) {
+ return Status::InternalError("paimon-rust read batch failed: {}",
+ consume_error(next.error));
+ }
+ // End of stream: both pointers are null.
+ if (next.batch.array == nullptr && next.batch.schema == nullptr) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ // RAII: the batch's Arrow release callbacks + container free run when
+ // `batch` leaves this scope, including on any early return.
+ ArrowBatch batch(next.batch);
+ arrow::Result<std::shared_ptr<arrow::RecordBatch>> import_result =
+ arrow::ImportRecordBatch(batch.array(), batch.schema());
+ if (!import_result.ok()) {
+ return Status::InternalError("failed to import paimon-rust arrow batch: {}",
+ import_result.status().message());
+ }
+ _pending_record_batch = std::move(import_result).ValueUnsafe();
+ _pending_record_batch_offset = 0;
+ if (_pending_record_batch->num_rows() == 0) {
+ _pending_record_batch.reset();
+ continue;
+ }
+ }
+
+ const size_t batch_size = static_cast<size_t>(
+ _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+ const size_t remaining = static_cast<size_t>(_pending_record_batch->num_rows()) -
+ _pending_record_batch_offset;
+ const size_t rows = std::min(batch_size, remaining);
+ RETURN_IF_ERROR(_fill_block_from_record_batch(_pending_record_batch, block,
+ _pending_record_batch_offset, rows));
+ _pending_record_batch_offset += rows;
+ if (_pending_record_batch_offset ==
+ static_cast<size_t>(_pending_record_batch->num_rows())) {
+ _pending_record_batch.reset();
+ _pending_record_batch_offset = 0;
+ }
+ _record_scan_rows(rows);
+ *eos = false;
+ return Status::OK();
+ }
+}
+
+Status PaimonRustTableReader::abort_split() {
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.close_timer);
+ _close_split_reader();
+ _split_eof = false;
+ }
+ return format::TableReader::abort_split();
+}
+
+#ifdef BE_TEST
+std::string PaimonRustTableReader::TEST_format_options(
+ const std::map<std::string, std::string>& options) {
+ return format_options(options);
+}
+
+std::map<std::string, std::string> PaimonRustTableReader::TEST_build_options(
+ TFileScanRangeParams* scan_params, const TFileRangeDesc& range) {
+ TFileScanRangeParams* previous_params = _scan_params;
+ TFileRangeDesc previous_range = _current_range;
+ _scan_params = scan_params;
+ _current_range = range;
+ std::map<std::string, std::string> options = _build_options();
+ _scan_params = previous_params;
+ _current_range = std::move(previous_range);
+ return options;
+}
+#endif
+
+Status PaimonRustTableReader::close() {
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.close_timer);
+ _close_split_reader();
+ _close_table();
+ }
+ return format::TableReader::close();
+}
+
+Status PaimonRustTableReader::_validate_rust_split(const TFileRangeDesc& range) const {
+ if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) {
+ return Status::InternalError(
+ "missing paimon_params for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ const auto& params = range.table_format_params.paimon_params;
+ if (!params.__isset.paimon_split || params.paimon_split.empty()) {
+ return Status::InternalError(
+ "missing paimon_split for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ if (params.__isset.reader_type && params.reader_type != TPaimonReaderType::PAIMON_RUST) {
+ return Status::InternalError(
+ "invalid reader_type for paimon rust reader, possibly caused by FE/BE protocol "
+ "mismatch");
+ }
+ if (!_resolve_table_path(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing paimon_table; cannot resolve paimon table location");
+ }
+ if (!_resolve_db_name(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing db_name; cannot open paimon table via schema json");
+ }
+ if (!_resolve_table_name(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing table_name; cannot open paimon table via schema json");
+ }
+ if (!_resolve_table_schema_json(range).has_value()) {
+ return Status::InternalError(
+ "paimon-rust missing paimon_table_schema_json; cannot open paimon table via "
+ "schema json");
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_open_split_reader(const TFileRangeDesc& range) {
+ // 1. Decode the FE-planned split first so we fail fast (and without any
+ // filesystem IO) when it is missing or malformed.
+ std::string split_bytes;
+ RETURN_IF_ERROR(_decode_split_bytes(&split_bytes));
+
+ // 2. Resolve identifier + table_path + FE-supplied TableSchema JSON.
+ auto table_path = _resolve_table_path(range).value();
+ auto db_name = _resolve_db_name(range).value();
+ auto table_name = _resolve_table_name(range).value();
+ auto schema_json = _resolve_table_schema_json(range).value();
+ auto branch_opt = _resolve_branch(range);
+
+ // 3. Assemble storage options: FE-supplied paimon options + hadoop_conf +
+ // OSS/S3 → AWS_* translations. These feed FileIO only (per
+ // paimon_table_from_schema_json contract); they are NOT merged into the
+ // supplied table schema.
+ auto options = _build_options();
+
+ auto opened_table_key =
+ std::make_tuple(table_path, schema_json, db_name, table_name, branch_opt, options);
+ if (!_handles || !_handles->table || _opened_table_key != opened_table_key) {
+ // A paimon scan reads one table, so the handle is opened at most once per
+ // distinct identity (e.g. re-created after a close); splits of the same
+ // table reuse it and only rebuild the read pipeline below.
+ _close_table();
+ _handles = std::make_unique<PaimonHandles>();
+
+ std::vector<paimon_option> c_options;
+ c_options.reserve(options.size());
+ for (const auto& kv : options) {
+ c_options.push_back(paimon_option {kv.first.c_str(), kv.second.c_str()});
+ }
+
+ LOG(INFO) << "paimon-rust opening table via schema json: db=" << db_name
+ << " table=" << table_name << " path=" << table_path
+ << " branch=" << (branch_opt.has_value() ? branch_opt.value() : "main")
+ << " storage_options=[" << format_options(options) << "]";
+
+ // Build the table directly from the FE-supplied schema JSON. The Rust
+ // side rejects null / empty branch, so we default to paimon's canonical
+ // "main" sentinel when FE did not set paimon_branch (i.e. the table is
+ // on the main branch — matches upstream Identifier.DEFAULT_MAIN_BRANCH).
+ const std::string& branch_str = branch_opt.has_value() ? branch_opt.value() : "main";
+ paimon_result_get_table tbl_res = paimon_table_from_schema_json(
+ table_path.c_str(), schema_json.c_str(), db_name.c_str(), table_name.c_str(),
+ branch_str.c_str(), c_options.empty() ? nullptr : c_options.data(),
+ c_options.size());
+ if (tbl_res.error != nullptr) {
+ return Status::InternalError(
+ "paimon-rust table_from_schema_json failed: db={} table={} err={}", db_name,
+ table_name, consume_error(tbl_res.error));
+ }
+ _handles->table.reset(tbl_res.table);
+ _opened_table_key = std::move(opened_table_key);
+ }
+
+ // 4. Build the read pipeline: read_builder -> case-insensitive -> projection.
+ // Bound the Rust Arrow allocation itself: wide rows must respect the scanner's
+ // adaptive probe size before they are materialized into a Doris block.
+ const auto batch_size = std::to_string(
+ _batch_size > 0 ? _batch_size : std::max(1, _runtime_state->batch_size()));
+ const paimon_option batch_option {"read.batch-size", batch_size.c_str()};
+ paimon_result_read_builder rb_res =
+ paimon_table_new_read_builder_with_options(_handles->table.get(), &batch_option, 1);
+ if (rb_res.error != nullptr) {
+ return Status::InternalError("paimon-rust new read builder failed: {}",
+ consume_error(rb_res.error));
+ }
+ _handles->read_builder.reset(rb_res.read_builder);
+
+ // Fold column casing on the Rust side so FE-normalized lowercase names
+ // resolve against tables with mixed-case column definitions.
+ if (paimon_error* case_err =
+ paimon_read_builder_with_case_sensitive(_handles->read_builder.get(), false)) {
+ return Status::InternalError("paimon-rust set case_sensitive failed: {}",
+ consume_error(case_err));
+ }
+
+ // Partition keys are excluded: they are materialized from split metadata
+ // (see _fill_non_arrow_columns), and paimon-rust does not emit them.
+ auto read_columns = _build_read_columns();
+ std::vector<const char*> projection;
+ projection.reserve(read_columns.size() + 1);
+ for (const auto& col : read_columns) {
+ projection.push_back(col.c_str());
+ }
+ projection.push_back(nullptr);
+ if (paimon_error* proj_err = paimon_read_builder_with_projection(_handles->read_builder.get(),
+ projection.data())) {
+ return Status::InternalError("paimon-rust set projection failed: {}",
+ consume_error(proj_err));
+ }
+
+ // Convert the scanner conjuncts into a paimon-rust filter and apply it.
+ RETURN_IF_ERROR(_apply_predicate());
+
+ // 5. Deserialize the FE-planned split into a one-split plan, so this
+ // scanner reads exactly the split it was assigned rather than replanning
+ // the whole table. The wire form is identical to what paimon-cpp consumes
+ // (`paimon::table::DataSplit::serialize`).
+ paimon_result_plan plan_res = paimon_plan_from_split_bytes(
+ reinterpret_cast<const uint8_t*>(split_bytes.data()), split_bytes.size());
+ if (plan_res.error != nullptr) {
+ return Status::InternalError("paimon-rust build plan failed: {}",
+ consume_error(plan_res.error));
+ }
+ _handles->plan.reset(plan_res.plan);
+
+ size_t num_splits = paimon_plan_num_splits(_handles->plan.get());
+ if (num_splits == 0) {
+ _split_eof = true;
+ return Status::OK();
+ }
+
+ // 6. Open the arrow stream over the plan.
+ paimon_result_new_read read_res = paimon_read_builder_new_read(_handles->read_builder.get());
+ if (read_res.error != nullptr) {
+ return Status::InternalError("paimon-rust new read failed: {}",
+ consume_error(read_res.error));
+ }
+ _handles->table_read.reset(read_res.read);
+
+ paimon_result_record_batch_reader rdr_res = paimon_table_read_to_arrow(
+ _handles->table_read.get(), _handles->plan.get(), /*offset=*/0, /*length=*/num_splits);
+ if (rdr_res.error != nullptr) {
+ return Status::InternalError("paimon-rust open arrow reader failed: {}",
+ consume_error(rdr_res.error));
+ }
+ _handles->reader.reset(rdr_res.reader);
+ return Status::OK();
+}
+
+void PaimonRustTableReader::_close_split_reader() {
+ _pending_record_batch.reset();
+ _pending_record_batch_offset = 0;
+ if (!_handles) {
+ return;
+ }
+ // Reverse of the declaration order in PaimonHandles.
+ _handles->reader.reset();
+ _handles->table_read.reset();
+ _handles->plan.reset();
+ _handles->read_builder.reset();
+}
+
+void PaimonRustTableReader::_close_table() {
+ if (!_handles) {
+ return;
+ }
+ _close_split_reader();
+ _handles->table.reset();
+ _opened_table_key.reset();
+}
+
+Status PaimonRustTableReader::_apply_predicate() {
+ if (_conjuncts.empty() || !_handles || !_handles->table || !_handles->read_builder) {
+ return Status::OK();
+ }
+ if (_scanner_profile != nullptr) {
+ COUNTER_UPDATE(_rust_predicates_input, _conjuncts.size());
+ for (const auto& conjunct : _conjuncts) {
+ if (conjunct && conjunct->root() && conjunct->root()->is_rf_wrapper()) {
+ COUNTER_UPDATE(_rust_runtime_filters_input, 1);
+ }
+ }
+ }
+ VLOG_DEBUG << "paimon-rust predicate pushdown: " << _conjuncts.size() << " conjunct(s) input";
+ // The conjunct VSlotRefs carry table global indices (positions), so the v2
+ // converter mode resolves fields by the projected column names; partition
+ // keys are excluded because the rust reader does not read them.
+ std::vector<std::string> names;
+ std::vector<DataTypePtr> types;
+ names.reserve(_projected_columns.size());
+ types.reserve(_projected_columns.size());
+ for (const auto& col : _projected_columns) {
+ if (col.is_partition_key) {
+ continue;
+ }
+ names.push_back(col.name);
+ types.push_back(col.type);
+ }
+ PaimonRustPredicateConverter converter(names, types, _handles->table.get());
+ paimon_predicate* predicate = converter.build(_conjuncts);
+ if (_scanner_profile != nullptr) {
+ COUNTER_UPDATE(_rust_predicates_converted, converter.converted_conjuncts());
+ }
+ if (predicate == nullptr) {
+ VLOG_DEBUG << "paimon-rust predicate pushdown: nothing convertible, no filter applied";
+ return Status::OK();
+ }
+ // paimon_read_builder_with_filter consumes the predicate (ownership moves to
+ // the builder) on every path, so we must not free it here.
+ if (paimon_error* err =
+ paimon_read_builder_with_filter(_handles->read_builder.get(), predicate)) {
+ return Status::InternalError("paimon-rust apply filter failed: {}", consume_error(err));
+ }
+ // Count application only after the C API accepts the filter; reader timers
+ // alone cannot distinguish pushdown from Doris residual-only execution.
+ if (_scanner_profile != nullptr) {
+ COUNTER_UPDATE(_rust_predicates_applied, converter.converted_conjuncts());
+ COUNTER_UPDATE(_rust_runtime_filters_applied, converter.converted_runtime_filters());
+ }
+ VLOG_DEBUG << "paimon-rust predicate pushdown: applied";
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::_fill_block_from_record_batch(
+ const std::shared_ptr<arrow::RecordBatch>& batch, Block* block, size_t start, size_t rows) {
+ SCOPED_TIMER(_rust_arrow_to_block_time);
+ DORIS_CHECK(batch != nullptr);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK_LE(start + rows, static_cast<size_t>(batch->num_rows()));
+ const auto arrow_start = static_cast<int64_t>(start);
+ const auto arrow_end = static_cast<int64_t>(start + rows);
+ std::unordered_set<size_t> materialized_indices;
+ materialized_indices.reserve(_projected_columns.size());
+ {
+ auto columns_guard = block->mutate_columns_scoped();
+ auto& columns = columns_guard.mutable_columns();
+ for (int c = 0; c < batch->num_columns(); ++c) {
+ const auto& field = batch->schema()->field(c);
+ if (field->name() == VALUE_KIND_FIELD) {
+ continue;
+ }
+ // Projected column names are FE-normalized to lowercase.
+ // paimon-rust's case_sensitive=false setting also case-folds column
+ // names in the schema output, so exact match works — but tolerate
+ // mixed-case Rust output by folding here as well.
+ auto it = _output_name_to_idx.find(field->name());
+ if (it == _output_name_to_idx.end()) {
+ it = _output_name_to_idx.find(to_lower(field->name()));
+ }
+ if (it == _output_name_to_idx.end()) {
+ // Skip columns that are not in the block (e.g. columns dropped by
+ // slot pruning).
+ continue;
+ }
+ const auto output_idx = it->second;
+ if (!materialized_indices.emplace(output_idx).second) {
+ return Status::InternalError("paimon-rust returned duplicate column '{}'",
+ field->name());
+ }
+ try {
+ RETURN_IF_ERROR(columns_guard.get_datatype_by_position(output_idx)
+ ->get_serde()
+ ->read_column_from_arrow(*columns[output_idx],
Review Comment:
Thanks. The original #66227 BE path also imports Rust Arrow batches directly into SerDe, so the sliced-BOOLEAN limitation predates this master port. Fixing the Arrow layout at the Rust/BE boundary deserves a focused change and a differential fixture; a blanket data-evolution fallback here would remove an original capability. Resolving as deferred, not fixed.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonRustReaderSelector.java:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.rest.RESTTokenFileIO;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.MultisetType;
+import org.apache.paimon.types.RowType;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Chooses the Rust reader only for table, split, storage and backend shapes it can read safely.
+ * Nested pruning and precision 7-9 timestamps still require JNI because the pinned Arrow ABI
+ * cannot preserve their full semantics. COUNT(*) metadata ranges bypass this selector entirely.
+ */
+final class PaimonRustReaderSelector {
+ static final String BACKEND_CAPABILITY = "paimon-rust-reader";
+ private static final String ENABLE_PAIMON_RUST_READER = "enable_paimon_rust_reader";
+ private static final String ENABLE_FILE_SCANNER_V2 = "enable_file_scanner_v2";
+ private static final String FORCE_JNI_SCANNER = "force_jni_scanner";
+ private static final String DELETION_VECTORS_MERGE_ON_READ = "deletion-vectors.merge-on-read";
+ private static final Set<String> VERIFIED_LOCATION_SCHEMES =
+ new HashSet<>(Arrays.asList("s3", "s3a", "oss", "hdfs", "file"));
+ private static final Set<String> VERIFIED_HDFS_OPTION_KEYS = new HashSet<>(Arrays.asList(
+ "fs.defaultFS",
+ "hadoop.security.authentication",
+ "hdfs.security.authentication",
+ "ipc.client.fallback-to-simple-auth-allowed"));
+
+ private final FileStoreTable table;
+ private final PaimonRustReaderCapabilities capabilities;
+ private final Map<String, String> backendStorageProperties;
+ private final boolean scanCompatible;
+ private final boolean partialUpdateOrAggregateWithDeletionVectors;
+ private final boolean deletionVectorsMergeOnRead;
+
+ PaimonRustReaderSelector(ConnectorSession session, boolean backendsSupportRust,
+ FileStoreTable table, List<ConnectorColumnHandle> columns,
+ Map<String, String> backendStorageProperties, boolean usesFallbackRead,
+ boolean incrementalRead, boolean projectedVariant) {
+ this.table = table;
+ this.backendStorageProperties = backendStorageProperties;
+ CoreOptions options = table.coreOptions();
+ CoreOptions.MergeEngine mergeEngine = options.mergeEngine();
+ partialUpdateOrAggregateWithDeletionVectors = options.deletionVectorsEnabled()
+ && (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+ || mergeEngine == CoreOptions.MergeEngine.AGGREGATE);
+ TableSchema schema = table.schema();
+ String mergeOnRead = schema.options().get(DELETION_VECTORS_MERGE_ON_READ);
+ deletionVectorsMergeOnRead = "true".equalsIgnoreCase(mergeOnRead);
+
+ String location = table.location().toString();
+ boolean enabled = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(ENABLE_PAIMON_RUST_READER));
+ boolean fileScannerV2 = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(ENABLE_FILE_SCANNER_V2));
+ boolean forceJniScanner = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(FORCE_JNI_SCANNER));
+ boolean restTokenTable = table.fileIO() instanceof RESTTokenFileIO;
+ boolean unsupportedMergeOption = (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+ || mergeEngine == CoreOptions.MergeEngine.AGGREGATE)
+ && hasRustUnsupportedMergeOption(schema.options(), mergeEngine);
+ boolean deduplicateIgnoreDelete = mergeEngine == CoreOptions.MergeEngine.DEDUPLICATE
+ && options.ignoreDelete();
+ String scanMode = PaimonScanParams.withoutTimeTravelSelectors(schema)
+ .options().get(CoreOptions.SCAN_MODE.key());
+ boolean scanModeSupported = scanMode == null || "default".equalsIgnoreCase(scanMode);
+
+ scanCompatible = enabled && fileScannerV2 && !forceJniScanner
+ && backendsSupportRust
+ && !usesFallbackRead && !incrementalRead && !projectedVariant
+ && !options.queryAuthEnabled() && !restTokenTable
+ && !(partialUpdateOrAggregateWithDeletionVectors && deletionVectorsMergeOnRead)
+ && !deduplicateIgnoreDelete
+ && !unsupportedMergeOption && scanModeSupported
+ && isRustVerifiedLocationScheme(location)
+ && (!isHdfsLocationScheme(location)
+ || isRustVerifiedHdfsBackend(backendStorageProperties))
+ && isCredentialProviderTranslatable(location, backendStorageProperties);
+ capabilities = new PaimonRustReaderCapabilities(table, columns);
+ }
+
+ boolean canRead(DataSplit split) {
+ if (!scanCompatible || splitHasExternalFiles(split)) {
+ return false;
+ }
+ if (splitHasOrcFile(split) && table.schema().fields().stream()
+ .anyMatch(field -> containsTimestampLtz(field.type()))) {
+ return false;
+ }
+ if (partialUpdateOrAggregateWithDeletionVectors && !deletionVectorsMergeOnRead
+ && !isFullyMaterializedPkDvSplit(split)) {
+ return false;
+ }
+ return capabilities.canRead(split);
+ }
+
+ static boolean isRustVerifiedLocationScheme(String location) {
+ if (location == null) {
+ return false;
+ }
+ int separator = location.indexOf("://");
Review Comment:
Thanks. I checked the original #66227 implementation: its location-scheme and HDFS eligibility checks use the same :// detection, so authority-less HDFS is an inherited edge case rather than a master-port regression. We will handle URI qualification/eligibility with a focused follow-up instead of changing this forward-port behavior. Resolving as deferred, not fixed.
##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn) \
+ struct type##_deleter { \
+ void operator()(paimon_##type* p) const { \
+ if (p) { \
+ freefn(p); \
+ } \
+ } \
+ }; \
+ using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+ explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+ ~ArrowBatch() {
+ auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+ auto* array = static_cast<ArrowArray*>(batch_.array);
+ if (array && array->release) {
+ array->release(array);
+ }
+ if (schema && schema->release) {
+ schema->release(schema);
+ }
+ paimon_arrow_batch_free(batch_);
+ }
+
+ ArrowBatch(const ArrowBatch&) = delete;
+ ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+ ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+ ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+ paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+ rust_handles::error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+ std::string out;
+ for (const auto& kv : options) {
+ if (!out.empty()) {
+ out += ", ";
+ }
+ out += kv.first;
+ }
+ return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+ rust_handles::table_ptr table;
+ rust_handles::read_builder_ptr read_builder;
+ rust_handles::plan_ptr plan;
+ rust_handles::table_read_ptr table_read;
+ rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+ RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+ {
+ // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+ // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.init_timer);
+ // Materialize TIMESTAMP_LTZ in the session timezone — the same
+ // convention as the JNI reader (PaimonJniScanner reads time_zone from
+ // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+ // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+ // of _ctz, so NTZ wall-clock semantics are preserved.
+ DORIS_CHECK(_runtime_state != nullptr);
+ _ctz = _runtime_state->timezone_obj();
+ if (_scanner_profile != nullptr) {
+ file_scan_profile::ensure_hierarchy(_scanner_profile);
+ _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+ file_scan_profile::TABLE_READER);
+ _rust_open_split_time =
+ ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+ _rust_read_batch_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+ _rust_arrow_to_block_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+ _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_converted = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+ }
+ // Projected column name -> fixed output position, registered with both the exact and
+ // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+ // semantics: exact match first, lower-case fallback on lookup).
+ _output_name_to_idx.reserve(_projected_columns.size() * 2);
+ for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+ _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+ _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+ }
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+ // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+ // are idempotent, and clear it only when a new split is explicitly prepared.
+ _close_split_reader();
+ _split_eof = false;
+ _current_range = options.current_range;
+ RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+ if (current_split_pruned()) {
+ return Status::OK();
+ }
+ if (_is_table_level_count_active()) {
+ // No rust pipeline is opened; get_block emits the synthetic count rows.
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.prepare_split_timer);
+ // Open may dominate the scan or fail before get_block; include it in the parent timer.
+ SCOPED_TIMER(_rust_total_time);
+ SCOPED_TIMER(_rust_open_split_time);
+ RETURN_IF_ERROR(_open_split_reader(options.current_range));
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.exec_timer);
+ SCOPED_TIMER(_rust_total_time);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK(eos != nullptr);
+ DORIS_CHECK(block->columns() == _projected_columns.size());
+ block->clear_column_data(_projected_columns.size());
+ *eos = false;
+
+ if (_is_table_level_count_active()) {
+ return _read_table_level_count(block, eos);
+ }
+
+ // num_splits == 0 yields an empty (but valid) stream: report EOF.
+ if (_split_eof) {
+ *eos = true;
+ return Status::OK();
+ }
+ if (!_handles || !_handles->reader) {
+ return Status::InternalError("paimon-rust reader is not initialized");
+ }
+
+ while (true) {
+ // Mirror the base TableReader cancellation contract so a cancelled query does not
+ // drain the whole split.
+ if (_io_ctx != nullptr && _io_ctx->should_stop) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ if (!_pending_record_batch) {
+ paimon_result_next_batch next;
+ {
+ SCOPED_TIMER(_rust_read_batch_time);
+ next = paimon_record_batch_reader_next(_handles->reader.get());
Review Comment:
Thanks for the finding. Rust-owned read/decode buffers are not yet charged to the Doris query tracker. This limitation is also present in the original branch-4.1 implementation. We are keeping this forward-port aligned with #66227 and will address cross-FFI memory accounting separately; resolving this thread as a documented follow-up, not as fixed in this PR.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonRustReaderSelector.java:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.rest.RESTTokenFileIO;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.MultisetType;
+import org.apache.paimon.types.RowType;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Chooses the Rust reader only for table, split, storage and backend shapes it can read safely.
+ * Nested pruning and precision 7-9 timestamps still require JNI because the pinned Arrow ABI
+ * cannot preserve their full semantics. COUNT(*) metadata ranges bypass this selector entirely.
+ */
+final class PaimonRustReaderSelector {
+ static final String BACKEND_CAPABILITY = "paimon-rust-reader";
+ private static final String ENABLE_PAIMON_RUST_READER = "enable_paimon_rust_reader";
+ private static final String ENABLE_FILE_SCANNER_V2 = "enable_file_scanner_v2";
+ private static final String FORCE_JNI_SCANNER = "force_jni_scanner";
+ private static final String DELETION_VECTORS_MERGE_ON_READ = "deletion-vectors.merge-on-read";
+ private static final Set<String> VERIFIED_LOCATION_SCHEMES =
+ new HashSet<>(Arrays.asList("s3", "s3a", "oss", "hdfs", "file"));
+ private static final Set<String> VERIFIED_HDFS_OPTION_KEYS = new HashSet<>(Arrays.asList(
+ "fs.defaultFS",
+ "hadoop.security.authentication",
+ "hdfs.security.authentication",
+ "ipc.client.fallback-to-simple-auth-allowed"));
+
+ private final FileStoreTable table;
+ private final PaimonRustReaderCapabilities capabilities;
+ private final Map<String, String> backendStorageProperties;
+ private final boolean scanCompatible;
+ private final boolean partialUpdateOrAggregateWithDeletionVectors;
+ private final boolean deletionVectorsMergeOnRead;
+
+ PaimonRustReaderSelector(ConnectorSession session, boolean backendsSupportRust,
+ FileStoreTable table, List<ConnectorColumnHandle> columns,
+ Map<String, String> backendStorageProperties, boolean usesFallbackRead,
+ boolean incrementalRead, boolean projectedVariant) {
+ this.table = table;
+ this.backendStorageProperties = backendStorageProperties;
+ CoreOptions options = table.coreOptions();
+ CoreOptions.MergeEngine mergeEngine = options.mergeEngine();
+ partialUpdateOrAggregateWithDeletionVectors = options.deletionVectorsEnabled()
+ && (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+ || mergeEngine == CoreOptions.MergeEngine.AGGREGATE);
+ TableSchema schema = table.schema();
+ String mergeOnRead = schema.options().get(DELETION_VECTORS_MERGE_ON_READ);
+ deletionVectorsMergeOnRead = "true".equalsIgnoreCase(mergeOnRead);
+
+ String location = table.location().toString();
+ boolean enabled = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(ENABLE_PAIMON_RUST_READER));
+ boolean fileScannerV2 = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(ENABLE_FILE_SCANNER_V2));
+ boolean forceJniScanner = session != null && "true".equalsIgnoreCase(
+ session.getSessionProperties().get(FORCE_JNI_SCANNER));
+ boolean restTokenTable = table.fileIO() instanceof RESTTokenFileIO;
+ boolean unsupportedMergeOption = (mergeEngine == CoreOptions.MergeEngine.PARTIAL_UPDATE
+ || mergeEngine == CoreOptions.MergeEngine.AGGREGATE)
+ && hasRustUnsupportedMergeOption(schema.options(), mergeEngine);
+ boolean deduplicateIgnoreDelete = mergeEngine == CoreOptions.MergeEngine.DEDUPLICATE
+ && options.ignoreDelete();
+ String scanMode = PaimonScanParams.withoutTimeTravelSelectors(schema)
+ .options().get(CoreOptions.SCAN_MODE.key());
+ boolean scanModeSupported = scanMode == null || "default".equalsIgnoreCase(scanMode);
+
+ scanCompatible = enabled && fileScannerV2 && !forceJniScanner
+ && backendsSupportRust
+ && !usesFallbackRead && !incrementalRead && !projectedVariant
+ && !options.queryAuthEnabled() && !restTokenTable
+ && !(partialUpdateOrAggregateWithDeletionVectors && deletionVectorsMergeOnRead)
+ && !deduplicateIgnoreDelete
+ && !unsupportedMergeOption && scanModeSupported
+ && isRustVerifiedLocationScheme(location)
+ && (!isHdfsLocationScheme(location)
+ || isRustVerifiedHdfsBackend(backendStorageProperties))
+ && isCredentialProviderTranslatable(location, backendStorageProperties);
+ capabilities = new PaimonRustReaderCapabilities(table, columns);
+ }
+
+ boolean canRead(DataSplit split) {
+ if (!scanCompatible || splitHasExternalFiles(split)) {
+ return false;
+ }
+ if (splitHasOrcFile(split) && table.schema().fields().stream()
Review Comment:
Thanks. The original #66227 selector likewise gates ORC TIMESTAMP_LTZ but not this low-precision TIMESTAMP range. This is a valid inherited limitation. We will address ORC timestamp decoding or a narrowly scoped eligibility rule separately, with parity coverage, while preserving the behavior of this forward-port. Resolving as deferred, not fixed.
##########
be/src/format_v2/table/paimon_rust_table_reader.cpp:
##########
@@ -0,0 +1,944 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/table/paimon_rust_table_reader.h"
+
+#include <algorithm>
+#include <utility>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "common/logging.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_const.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "format_v2/column_mapper.h"
+#include "format_v2/table/paimon_rust_predicate_converter.h"
+#include "runtime/descriptors.h"
+#include "runtime/file_scan_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/string_util.h"
+#include "util/timezone_utils.h"
+#include "util/url_coding.h"
+
+extern "C" {
+#include "paimon_rust/paimon.h"
+}
+
+namespace doris::format::paimon {
+
+namespace rust_handles {
+
+// ---------------------------------------------------------------------------
+// RAII wrappers over the paimon-rust C handles. Each handle is an opaque
+// pointer owned by Rust and released by a matching paimon_*_free function.
+// ---------------------------------------------------------------------------
+#define PAIMON_OWNED(type, freefn) \
+ struct type##_deleter { \
+ void operator()(paimon_##type* p) const { \
+ if (p) { \
+ freefn(p); \
+ } \
+ } \
+ }; \
+ using type##_ptr = std::unique_ptr<paimon_##type, type##_deleter>
+
+PAIMON_OWNED(table, paimon_table_free);
+PAIMON_OWNED(read_builder, paimon_read_builder_free);
+PAIMON_OWNED(plan, paimon_plan_free);
+PAIMON_OWNED(table_read, paimon_table_read_free);
+PAIMON_OWNED(record_batch_reader, paimon_record_batch_reader_free);
+PAIMON_OWNED(error, paimon_error_free);
+
+#undef PAIMON_OWNED
+
+} // namespace rust_handles
+
+namespace {
+constexpr const char* VALUE_KIND_FIELD = "_VALUE_KIND";
+
+// One Arrow batch (schema + array containers). Owning it requires a two-step
+// teardown that the unique_ptr deleters above can't express: first invoke the
+// Arrow C Data Interface `release` callback on each struct (hands buffers back
+// to the producer), then free the container structs via paimon_arrow_batch_free.
+class ArrowBatch {
+public:
+ explicit ArrowBatch(paimon_arrow_batch batch) : batch_(batch) {}
+ ~ArrowBatch() {
+ auto* schema = static_cast<ArrowSchema*>(batch_.schema);
+ auto* array = static_cast<ArrowArray*>(batch_.array);
+ if (array && array->release) {
+ array->release(array);
+ }
+ if (schema && schema->release) {
+ schema->release(schema);
+ }
+ paimon_arrow_batch_free(batch_);
+ }
+
+ ArrowBatch(const ArrowBatch&) = delete;
+ ArrowBatch& operator=(const ArrowBatch&) = delete;
+
+ ArrowSchema* schema() const { return static_cast<ArrowSchema*>(batch_.schema); }
+ ArrowArray* array() const { return static_cast<ArrowArray*>(batch_.array); }
+
+private:
+ paimon_arrow_batch batch_;
+};
+
+// Render a paimon_error into a string. Takes ownership of `err` via RAII so it
+// is freed on every return path. Safe to call with nullptr.
+std::string consume_error(paimon_error* err) {
+ rust_handles::error_ptr owned(err);
+ if (!owned) {
+ return "unknown error";
+ }
+ std::string msg;
+ if (owned->message.data != nullptr && owned->message.len > 0) {
+ msg.assign(reinterpret_cast<const char*>(owned->message.data), owned->message.len);
+ }
+ return "code=" + std::to_string(owned->code) + ", msg=" + msg;
+}
+
+// Render storage option KEYS for diagnostics. Values are never rendered:
+// credential keys arrive under many spellings and cases (AWS_SECRET_KEY,
+// AWS_TOKEN, fs.oss.accessKeySecret, s3.secret-key, ...), and a key-name
+// blocklist that misses one alias leaks the value into the INFO log, so
+// only the key names are printed at all.
+std::string format_options(const std::map<std::string, std::string>& options) {
+ std::string out;
+ for (const auto& kv : options) {
+ if (!out.empty()) {
+ out += ", ";
+ }
+ out += kv.first;
+ }
+ return out;
+}
+
+} // namespace
+
+// Paimon-rust handles. Order of members matters: destruction runs in reverse
+// declaration order, and the read_builder depends on the table while the arrow
+// reader depends on the whole pipeline above it. So the table MUST be declared
+// first (destroyed last) and the record batch reader last.
+struct PaimonRustTableReader::PaimonHandles {
+ rust_handles::table_ptr table;
+ rust_handles::read_builder_ptr read_builder;
+ rust_handles::plan_ptr plan;
+ rust_handles::table_read_ptr table_read;
+ rust_handles::record_batch_reader_ptr reader;
+};
+
+PaimonRustTableReader::PaimonRustTableReader() = default;
+
+PaimonRustTableReader::~PaimonRustTableReader() = default;
+
+Status PaimonRustTableReader::init(format::TableReadOptions&& options) {
+ RETURN_IF_ERROR(format::TableReader::init(std::move(options)));
+ {
+ // Base and derived scopes must not overlap on the same counter: RuntimeProfile timers
+ // add deltas, so nested use would double-count instead of extending lifecycle coverage.
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.init_timer);
+ // Materialize TIMESTAMP_LTZ in the session timezone — the same
+ // convention as the JNI reader (PaimonJniScanner reads time_zone from
+ // its scan params) and lance_reader. Timezone-naive (paimon TIMESTAMP)
+ // arrow values are decoded in UTC by the DateTimeV2 serde regardless
+ // of _ctz, so NTZ wall-clock semantics are preserved.
+ DORIS_CHECK(_runtime_state != nullptr);
+ _ctz = _runtime_state->timezone_obj();
+ if (_scanner_profile != nullptr) {
+ file_scan_profile::ensure_hierarchy(_scanner_profile);
+ _rust_total_time = ADD_CHILD_TIMER(_scanner_profile, "PaimonRustReader",
+ file_scan_profile::TABLE_READER);
+ _rust_open_split_time =
+ ADD_CHILD_TIMER(_scanner_profile, "OpenSplitTime", "PaimonRustReader");
+ _rust_read_batch_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ReadBatchTime", "PaimonRustReader");
+ _rust_arrow_to_block_time =
+ ADD_CHILD_TIMER(_scanner_profile, "ArrowToBlockTime", "PaimonRustReader");
+ _rust_predicates_input = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesInput",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_converted = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustPredicatesConverted", TUnit::UNIT, "PaimonRustReader");
+ _rust_predicates_applied = ADD_CHILD_COUNTER(_scanner_profile, "RustPredicatesApplied",
+ TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_input = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersInput", TUnit::UNIT, "PaimonRustReader");
+ _rust_runtime_filters_applied = ADD_CHILD_COUNTER(
+ _scanner_profile, "RustRuntimeFiltersApplied", TUnit::UNIT, "PaimonRustReader");
+ }
+ // Projected column name -> fixed output position, registered with both the exact and
+ // the lower-case spelling so mixed-case Rust schema output still resolves (v1
+ // semantics: exact match first, lower-case fallback on lookup).
+ _output_name_to_idx.reserve(_projected_columns.size() * 2);
+ for (size_t idx = 0; idx < _projected_columns.size(); ++idx) {
+ _output_name_to_idx.emplace(_projected_columns[idx].name, idx);
+ _output_name_to_idx.emplace(to_lower(_projected_columns[idx].name), idx);
+ }
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::prepare_split(const format::SplitReadOptions& options) {
+ // EOF belongs to the previous split. Keep it set after closing that split so repeated reads
+ // are idempotent, and clear it only when a new split is explicitly prepared.
+ _close_split_reader();
+ _split_eof = false;
+ _current_range = options.current_range;
+ RETURN_IF_ERROR(format::TableReader::prepare_split(options));
+ if (current_split_pruned()) {
+ return Status::OK();
+ }
+ if (_is_table_level_count_active()) {
+ // No rust pipeline is opened; get_block emits the synthetic count rows.
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(_validate_rust_split(options.current_range));
+ {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.prepare_split_timer);
+ // Open may dominate the scan or fail before get_block; include it in the parent timer.
+ SCOPED_TIMER(_rust_total_time);
+ SCOPED_TIMER(_rust_open_split_time);
+ RETURN_IF_ERROR(_open_split_reader(options.current_range));
+ }
+ return Status::OK();
+}
+
+Status PaimonRustTableReader::get_block(Block* block, bool* eos) {
+ SCOPED_TIMER(_profile.total_timer);
+ SCOPED_TIMER(_profile.exec_timer);
+ SCOPED_TIMER(_rust_total_time);
+ DORIS_CHECK(block != nullptr);
+ DORIS_CHECK(eos != nullptr);
+ DORIS_CHECK(block->columns() == _projected_columns.size());
+ block->clear_column_data(_projected_columns.size());
+ *eos = false;
+
+ if (_is_table_level_count_active()) {
+ return _read_table_level_count(block, eos);
+ }
+
+ // num_splits == 0 yields an empty (but valid) stream: report EOF.
+ if (_split_eof) {
+ *eos = true;
+ return Status::OK();
+ }
+ if (!_handles || !_handles->reader) {
+ return Status::InternalError("paimon-rust reader is not initialized");
+ }
+
+ while (true) {
+ // Mirror the base TableReader cancellation contract so a cancelled query does not
+ // drain the whole split.
+ if (_io_ctx != nullptr && _io_ctx->should_stop) {
+ _split_eof = true;
+ _close_split_reader();
+ *eos = true;
+ return Status::OK();
+ }
+
+ if (!_pending_record_batch) {
+ paimon_result_next_batch next;
+ {
+ SCOPED_TIMER(_rust_read_batch_time);
+ next = paimon_record_batch_reader_next(_handles->reader.get());
+ }
+ if (next.error != nullptr) {
Review Comment:
Thanks. The original #66227 Rust reader also maps read errors to INTERNAL_ERROR, so missing-file classification is inherited. A correct fix needs a distinguishable missing-data-file error from the Rust binding; converting generic IO errors to NOT_FOUND would be unsafe. We will pursue that separately with differential coverage. Resolving as deferred, not fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) Support Paimon Rust reader on master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68825:
URL: https://github.com/apache/doris/pull/68825#issuecomment-6187718461
# FE Regression Coverage Report
Increment line coverage ` 20.00% (19/95)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68825_b3e30fa9240264b19d185ee444c6531d4106a8ff_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org